基础知识

所以基本上我已经编写了一个程序，可以在 Node 中为 MongoDB 生成测试数据。

问题

为此，程序读取模式文件并从中生成指定数量的测试数据。问题是这些数据最终会变得非常大(考虑创建 1M 用户(具有它需要的所有属性)和 20M 聊天消息(使用 userFrom 和 userTo)并且它必须将所有这些保存在 RAM 中以对其进行修改/转换/映射，然后将其保存到文件中。

工作原理

程序是这样运行的:

1. 读取模式文件
2. 从模式创建测试数据并将其存储在一个结构中(请往下看结构)
3. 运行此结构并将所有对象 referenceTo 链接到具有匹配 referenceKey 的随机对象。
4. 转换MongoDB插入语句的string[]中的对象结构
5. 将 string[] 存储在一个文件中。

这是生成的测试数据的结构:

export interface IGeneratedCollection {
    dbName: string,                 // Name of the database
    collectionName: string,         // Name of the collection
    documents: IGeneratedDocument[] // One collection has many documents
}

export interface IGeneratedDocument {
    documentFields: IGeneratedField [] // One document has many fields (which are recursive, because of nested documents)
}

export interface IGeneratedField {
    fieldName: string, // Name of the property
    fieldValue: any,   // Value of the property (Can also be IGeneratedField, IGeneratedField[], ...)
    fieldNeedsQuotations?: boolean, // If the Value needs to be saved with " ... "
    fieldIsObject?: boolean,        // If the Value is a object (stored as IGeneratedField[]) (To handle it different when transforming to MongoDB inserts)
    fieldIsJsonObject?: boolean,    // If the Value is a plain JSON object
    fieldIsArray?: boolean,         // If the Value is array of objects (stored as array of IGeneratedField[])
    referenceKey?: number,          // Field flagged to be a key
    referenceTo?: number            // Value gets set to a random object with matching referenceKey
}

实际数据

因此在具有 1M 用户和 20M 消息的示例中，它看起来像这样:

• 1x IGeneratedCollection(collectionName = "users")
  • 1Mx IGeneratedDocument
    • 10x IGeneratedField(例如每个用户有 10 个字段)
• 1x IGeneratedCollection(collectionName = "messages")
  • 20Mx IGeneratedDocument
    • 3x IGeneratedField(消息、userFrom、userTo)

这将产生 190M 个 IGeneratedField 实例(1x1Mx10 + 1x20Mx3x = 190M)。

结论

对于 RAM 来说，这显然需要处理很多事情，因为它需要同时存储所有这些。

Temporary Solution

It now works like that:

1. Generate 500 documents(rows in sql) at a time
2. JSON.stringify those 500 documents and put them in a SQLite table with the schema (dbName STRING, collectionName STRING, value JSON)
3. Remove those 500 documents from JS and let the Garbage Collector do its thing
4. Repeat until all data is generated and in the SQLite table
5. Take one of the rows (each containing 500 documents) at a time, apply JSON.parse and search for keys in them
6. Repeat until all data is queried and all keys retrieved
7. Take one of the rows at a time, apply JSON.parse and search for key references in them
8. Apply JSON.stringify and update the row if necessary (if key references found and resolved)
9. Repeat until all data is queried and all keys are resolved
10. Take one of the rows at a time, apply JSON.parse and transform the documents to valid sql/mongodb inserts
11. Add the insert (string) in a SQLite table with the schema (singleInsert STRING)
12. Remove the old and now unused row from the SQLite table
13. Write all inserts to file (if run from the command line) or return a dataHandle to query the data in the SQLite table (if run from other node app)

This solution does handle the problem with RAM, because SQLite automatically swaps to the Harddrive when the RAM is full

BUT

As you can see there are a lot of JSON.parse and JSON.stringify involved, which slows down the whole process drastically

What I have thought:

Maybe I should modify the IGeneratedField to only use shortend names as variables (fieldName -> fn, fieldValue -> fv, fieldIsObject -> fio, fieldIsArray -> fia, ....)

This would make the needed storage in the SQLite table smaller, BUT it would also make the code harder to read

Use a document oriented database (But I have not really found one), to handle JSON data better

问题

有没有更好的解决方案来处理 Node 中这样的大对象？

我的临时解决方案可以吗？这有什么不好？是否可以对其进行更改以使其性能更好？

最佳答案

从概念上讲，在流中生成项目。

您不需要 db 中的所有 1M 用户。您可以一次添加 10k。

对于消息，从数据库中随机抽取 2n 个用户，这些用户相互发送消息。重复直到满意为止。

例子:

// Assume Users and Messages are both db.collections
// Assume functions generateUser() and generateMessage(u1, u2) exist.
const desiredUsers = 10000;
const desiredMessages = 5000000;
const blockSize = 1000;


(async () => {

for (const i of _.range(desiredUsers / blockSize) ) {
    const users = _.range(blockSize).map(generateUser);
    await Users.insertMany(users);
}


for (const i of _.range(desiredMessages / blockSize) ) {
    const users = await Users.aggregate([ { $sample: { size: 2 * blockSize } } ]).toArray();
    const messages = _.chunk(users, 2).map( (usr) => generateMessage(usr[0], usr[1]));
    await Messages.insertMany(messages);
}

})();

根据您调整流的方式，您会得到不同的分布。这是均匀分布。您可以通过交错用户和消息来获得更多的长尾分布。例如，您可能希望对留言板执行此操作。

在我将 blockSize 切换为 1000 后变为 200MB。

关于javascript - 如何在 node.js 中存储和修改大型数据集？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/56519516/