Commit 04039187 authored by Michael Henretty's avatar Michael Henretty

load all txt files in server/data for sentences

parent 6c2e5f9d
This diff is collapsed.
He wants me to write down only things that I think about, but not things I plan to do.
I was wondering if you would work on this project with me because I can’t find anyone else.
Drawing from my own experience as a learner of English and German, I value engaging activities that involve everyday conversation.
But it is not a class that was requested by our alumni or prospective students.
This will help you effectively collaborate with developers and coordinate cross-team projects throughout your career.
After all, who doesn’t want to overcome new challenges and achieve great heights?
Maybe he should not call people names that they don't want to be called.
I told him we could teach her to ignore people who waste her time.
I am definitely open to the idea because I’ve always wanted to work as a small business owner.
I don’t mind working from home most of the time, but I do miss having coworkers.
Mary told me that she got to meet up with you while she was back in San Francisco.
We miss you and miss having a friend like you, and I am so happy that you two got to catch up.
We have also started contemplating the possibility of moving back to Colorado.
I was wondering if you could tell me a bit more about what it's like to live and work there.
A part of me is also trying to come to terms with the fact that I won't be able to travel as much anymore.
I was just in town earlier this year, so it is kind of crazy to think about what took place right after.
You can get around very easily by train — which is a very relaxing and affordable way to travel.
Suddenly, there is so much more time in a day because she no longer has to spend hours commuting.
She keeps saying that she has no regrets about moving back home, but I’m not sure if it’s entirely true.
I think it's about bird hunting, but now I can see how that might be misinterpreted.
He has done so much ghostwriting that he starts to feel like a ghost, too.
Yeah, it’s definitely worth checking out, especially if the pay is reasonable and the work is good for your career. There was absolutely no reason for her to announce it to the entire company.
He made us stand and watch him paint for like an hour.
And then he would ask us to go back to our easels and replicate all the techniques in one go.
It was impossible to remember all the details that he covered in an hour.
Earlier this year, we found a really nice place near my office, and we moved in together.
I have been wanting to go and waiting for him to go with me.
I finally went by myself and came back this morning feeling happy and refreshed.
They can be a museum, a nearby town, a park, or any place that I have never been before.
He dabbled in painting, drawing, and photography before finding his true passion.
She got a drawing pad and began dedicating a few days a week to sketching her surroundings.
She loves to read, but is unsure how reading is helping her become more creative.
I hate to think that I will have to leave, but I guess that’s not really up to me.
Not that I was listening, but I noticed that you sounded really stressed when you were explaining the situation to him.
I'm looking forward to having more stability in my position and being done with the whole process.
Maybe we can wait a little bit before changing the titles and the text everywhere else?
I know this might sound really bad, but listening to this music just makes me want to go hide in the mountains.
I am already imagining myself at a cafe somewhere far away, scribbling away and daydreaming about the past.
I am upset that they get to have all the time in the world while acting like they earned it.
Work has been rough in the last few months because of a major project launch.
She said she was so exhausted and unhappy that she thought of leaving, and she did.
He thought it was normal to panic and to feel as though he really didn’t deserve anything good or great.
I would like to take a nice long break with no financial repercussions, but we all know that’s not going to happen.
I have to admit that my horizon has expanded exponentially.
I feel grateful for the time that we have spend together.
Yeah, better if people are already familiar with one and are thinking of switching to the other.
I think it’s too overwhelming to introduce two complex tools that basically do the same things.
I haven’t seen her in ages, and I hope she remembers me still!
Some people prefer not to quantify their feelings that way.
It is rare that I get to sit down and write during the day.
It’s important to keep an open mind, and she has urged me to do the same.
It always helps to be learning something new and exciting.
The fire alarm went off again, but when I looked outside, there was no one there.
Everyone is just standing by the building, waiting to get in.
I wonder how many times a year this happens.
She seems very indifferent, almost apathetic.
She told me she feels hopeful and optimistic.
They are not really friends, they just work together and talk from time to time.
Traveling alone is good for meeting new people.
When talking to someone who has just started to learn the language, it’s good to speak slowly and clearly.
She has always been very kind and friendly to everyone, even strangers.
This source diff could not be displayed because it is too large. You can view the blob instead.
import * as http from 'http';
import WebHook from './webhook';
const SENTENCE_FILE = '../../data/new-sentences.txt';
const path = require('path');
const fs = require('fs');
const Promise = require('bluebird');
const SENTENCE_FOLDER = '../../data/';
export default class API {
sentencesCache: String[];
webhook: WebHook;
constructor() {
this.webhook = new WebHook();
this.getSentences();
}
private getSentenceFolder() {
return path.join(__dirname, SENTENCE_FOLDER);
}
private getRandomSentences(count: number): Promise<string[]> {
......@@ -25,6 +30,33 @@ export default class API {
});
}
private getFilesInFolder(folderpath) {
return new Promise((res, rej) => {
fs.readdir(folderpath, (err, files) => {
if (err) {
rej(err);
return;
}
res(files);
});
});
}
private getFileContents(filepath) {
return new Promise((res, rej) => {
fs.readFile(filepath, {
contents: 'utf8'
}, (err, data) => {
if (err) {
rej(err);
return;
}
res(data.toString());
});
});
}
/**
* Is this request directed at the api?
*/
......@@ -61,23 +93,39 @@ export default class API {
return Promise.resolve(this.sentencesCache);
}
return new Promise((resolve: Function, reject: Function) => {
let sentencePath = path.join(__dirname, SENTENCE_FILE);
let contents = fs.readFileSync(sentencePath, {
encoding: 'utf8'
});
return this.getFilesInFolder(this.getSentenceFolder())
.then(files => {
return Promise.all(files.map(filename => {
let sentences = contents.split('\n');
// TODO: Spaces are used to mark paragraphs, ignore them for now.
sentences = sentences.filter(s => s.length);
this.sentencesCache = sentences;
if (this.sentencesCache.length < 10) {
reject('not enough sentences');
return;
}
// Only parse the top-level text files, not any sub folders.
if (filename.split('.').pop() !== 'txt') {
return null;
}
resolve(this.sentencesCache);
});
let filepath = path.join(this.getSentenceFolder(), filename);
return this.getFileContents(filepath);
}));
})
// Chop the array of content strings into an array of sentences.
.then((values) => {
let sentences = [];
let sentenceArrays = values.map(fileContents => {
if (!fileContents) {
return [];
}
return fileContents.split('\n');
});
sentences = sentences.concat.apply(sentences, sentenceArrays);
console.log('sentences loaded', sentences.length);
this.sentencesCache = sentences;
})
.catch(err => {
console.error('could not retrieve sentences', err);
});
}
/**
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment