Loading... <h2>NodeBB是什么?</h2><p>NodeBB是一个开源的论坛软件,基于Node.JS开发,使用Redis或MongoDB来存储数据(详见 <span class="external-link"><a class="no-external-link" href="https://nodebb.org/" target="_blank">https://nodebb.org/<i data-feather='external-link'></i></a></span> )</p><h2>为什么要迁移数据?</h2><p>比起MongoDB,Redis更适合用来存储缓存而不是当成数据库来用(参考<span class="external-link"><a class="no-external-link" href="https://www.cnblogs.com/chinesern/p/5581422.html" target="_blank">这篇文章<i data-feather='external-link'></i></a></span>)</p><p>NodeBB虽然可以使用不同的数据库来存储数据,但是官方并没有提供把数据从一种数据库迁移到另一种数据库的工具,于是我写了一个(</p><!--more--><h2>下面开始正片</h2><p>首先我们需要知道,NodeBB使用两种数据库存储时,数据的结构分别是什么以及他们的区别。<br>搭建两个NodeBB站点,分别使用Redis和MongoDB作为数据库(搭建步骤略过,可以参考官方文档或者<span class="external-link"><a class="no-external-link" href="https://docs.nodebb-cn.org/336022" target="_blank">中文社区的文档<i data-feather='external-link'></i></a></span>),然后在这两个站点进行一些注册/发帖等操作,让数据库里存储一些数据<br>(我这里有现成的NodeBB站点,所以就不用搭建新的来演示了)<br>现在我们连接到数据库来看看里面有什么东西</p><p>使用<span class="external-link"><a class="no-external-link" href="https://redisdesktop.com" target="_blank">Redis Desktop Manager<i data-feather='external-link'></i></a></span>连接到Redis Server<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5a0c762ab0.png" alt="Redis" title="Redis" /></p><p>使用<span class="external-link"><a class="no-external-link" href="https://www.mongodb.com/products/compass" target="_blank">MongoDB Compass<i data-feather='external-link'></i></a></span>连接到MongoDB<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5a1783bb1c.png" alt="Mongo" title="Mongo" /></p><p>对比两个数据库中的内容,不难发现:</p><ul><li>NodeBB在MongoDB中也是按类似于Redis的方式来存储数据的</li><li>Redis中的key对应MongoDB中Document的 <code>_key</code> 字段</li><li><p>对于Redis的5种数据类型,在MongoDB中对应的Document分别是</p><ul><li><p>字符串(String):</p><pre><code class="lang-json">{ _key: Redis中的key, data: 字符串的值 }</code></pre></li><li><p>哈希(Hash): Redis的Hash和MongoDB的Document结构相似,除了 <code>_key</code> 字段以外基本相同。</p><ul><li><p>例1:这样的一个Hash:</p><pre><code>127.0.0.1:6379> HGETALL user:1 1) "username" 2) "BBleae" 3) "email" 4) "bbleae@baka.studio"</code></pre><p>在MongoDB中对应的Document为</p><pre><code class="lang-json">{ _key: "user:1", username:"BBleae", email:"bbleae@baka.studio" }</code></pre></li><li>例2:<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5b04e3033b.png" alt="hash" title="hash" /></li></ul></li><li>列表(List):<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5b0ce92962.png" alt="list" title="list" /></li><li>集合(Set):<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5b25046927.png" alt="set" title="set" /></li><li>有序集合(sorted set):<br><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAABS2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxMzggNzkuMTU5ODI0LCAyMDE2LzA5LzE0LTAxOjA5OjAxICAgICAgICAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8+CiA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgo8P3hwYWNrZXQgZW5kPSJyIj8+IEmuOgAAAA1JREFUCJljePfx038ACXMD0ZVlJAYAAAAASUVORK5CYII=" data-original="https://i.loli.net/2018/10/28/5bd5b3b377146.png" alt="zset" title="zset" /></li></ul></li></ul><h2>代码:</h2><p>用Node.JS实现数据的迁移(有一些<strong>很重要的</strong>细节后面会解释)</p><pre><code class="lang-javascript">'use strict' var Promise = require('bluebird') var MongoClient = require('mongodb').MongoClient var redis = require('redis') Promise.promisifyAll(redis) var redisClient = redis.createClient({ host: '127.0.0.1', port: '6379' }) async function start() { const url = 'mongodb://localhost:27017' const dbName = 'nodebb' const client = new MongoClient(url) try { await client.connect() var col = client.db(dbName).collection('objects') var keys = await redisClient.keysAsync('*') console.log('total keys:', keys.length) var inserted = 0 for (const i of keys) { if (i.indexOf('sess:') == -1 && i.indexOf('nodebbpostsearch:') == -1 && i.indexOf('nodebbtopicsearch:') == -1) { const type = await redisClient.typeAsync(i) switch (type) { case 'hash': let hash = await redisClient.hgetallAsync(i) for (const prop in hash) { if (Number(hash[prop]) != 0 && !Number(hash[prop])) hash[prop] = Number(hash[prop]) } hash._key = i await col.insertOne(hash) console.log(++inserted) break case 'zset': let zset = await redisClient.zrangeAsync(i, 0, -1, 'withscores') let zlen = zset.length for (let j = 0; j < zlen; j += 2) { await col.insertOne({ _key: i, value: zset[j], score: Number(zset[j + 1]) }) } console.log(++inserted) break case 'set': let set = await redisClient.smembersAsync(i) await col.insertOne({ _key: i, members: set }) console.log(++inserted) break case 'string': let str = await redisClient.getAsync(i) await col.insertOne({ _key: i, data: str }) console.log(++inserted) break case 'list': let list = await redisClient.lrangeAsync(i, 0, -1) await col.insertOne({ _key: i, array: list }) console.log(++inserted) break default: console.log("Unknown key type:", i) break } } } } catch (err) { console.log(err.stack) } client.close() } start()</code></pre><p>一些细节的说明:</p><pre><code>if (i.indexOf('sess:') == -1 && i.indexOf('nodebbpostsearch:') == -1 && i.indexOf('nodebbtopicsearch:') == -1)</code></pre><p>忽略所有以sess:/nodebbpostsearch:/nodebbtopicsearch:开头的key<br>sess:开头的是Session存储,NodeBB可以单独使用Redis存储Session,而且即使用MongoDB来存储,也会存储在单独的collection里(而不是名为objects的collection!)<br>nodebbpostsearch:和nodebbtopicsearch:是NodeBB自带插件 <code>nodebb-plugin-dbsearch</code> 使用的数据,这部分如果用MongoDB存储也会在单独的collection里,另外这部分数据可以在网站后台Reindex重新生成</p><pre><code>if (Number(hash[prop]) != 0 && !Number(hash[prop])) hash[prop] = Number(hash[prop])</code></pre><pre><code>await col.insertOne({ _key: i, value: zset[j], score: Number(zset[j + 1])</code></pre><p><strong>这两段代码非常重要!</strong> 它们把字符串转换为数字,因为Redis里存储的都是字符串,而如果使用MongoDB存储,有些地方会用到 <code>$inc</code> 这个操作符,对字符串使用 <code>$inc</code> 毫无疑问会报错!</p> Last modification:October 31st, 2018 at 09:59 pm © 允许规范转载