Dbs

Mongodb error solutions

Mongodb error solutions

Mongodb error solutions

suo.li
Mongodb 报错案例 案例1 command [root@xxxxx]# /opt/mongodb-3.2.10/bin/mongoimport --host xxxxx --port 20000 --collection user_week_activity --db cake --type json --upsert --file /home/suo.li/result.json -vv error using upsert fields: [_id] 2021-07-23T17:34:59.801+0800 using 2 decoding workers 2021-07-23T17:34:59.801+0800 using 1 insert workers 2021-07-23T17:34:59.801+0800 filesize: 4611622 bytes 2021-07-23T17:34:59.801+0800 using fields: 2021-07-23T17:34:59.804+0800 connected to: xxxxx:20000 2021-07-23T17:34:59.804+0800 ns: cake.user_week_activity 2021-07-23T17:34:59.804+0800 connected to node type: mongos 2021-07-23T17:34:59.804+0800 using write concern: w='majority', j=false, fsync=false, wtimeout=0 2021-07-23T17:34:59.806+0800 using write concern: w='majority', j=false, fsync=false, wtimeout=0 2021-07-23T17:34:59.806+0800 Failed: error processing document #1: invalid character 'ÿ' looking for beginning of value 2021-07-23T17:34:59.806+0800 imported 0 documents cause 由于result.json是m
Mongodb Shard集群手动将大块的jumbo chunk拆分

Mongodb Shard集群手动将大块的jumbo chunk拆分

Mongodb Shard Manual Split Jumbo Chunk

suo.li
最近压测MongoDB Shard集群时发现集群中四个shard 的shard1的primary机器负载异常的高,其它三个shard 的primary负载在40%左右,shard1的primary负载就已经 飙升超过80%,经过查看日志发现shar
Mongodb Shard集群中的复制集主备切换

Mongodb Shard集群中的复制集主备切换

Mongodb shard active-Standby switch

suo.li
最近公司准备上线新的业务,需要压测MongoDB Shard集群采用多个机器来回放日志进行压测, 使用js解析以往的日志,并拼装成新的请求来请求服务,进行压测。使用25台机器并发发起 请求,每分钟20000个请求,MongoDB Shard集群的p
将PG数据库中的表数据导出到MongoDB

将PG数据库中的表数据导出到MongoDB

Export data from postgresql to mongodb

suo.li
1. PG sql 使用row_to_json()函数将数据转为json格式 使用COPY命令将数据写到标准输出 注意:使用COPY命令直接将表数据写到某个文件中需要superuser权限 1 > select regextappid as appid, bindings->'devid' as devid, bindings->'wechat' as wechat from user_info; 2. > SELECT row_to_json(sso_bindings) FROM ( select regextappid as appid, bindings->'devid' as devid, bindings->'wechat' as wechat from user_info ) sso_bindings; 3.
Memcached

Memcached

Memcached

suo.li
memcached 简介 memcache 本身没有像 redis 所具备的数据持久化功能,比如 RDB 和 AOF 都没有,但是 可以通过做集群同步的方式,让各 memcache 服务器的数据进行同步,从而实现数据的一致 性,即保证各 memcache 的数据是一样的,即使有任何一台 memcache 发生故障,只要集 群种有一台 memcache 可用就不会出现数据丢失,当其
pgxc_ctl部署postgresql-xl集群

pgxc_ctl部署postgresql-xl集群

using pgxc_ctl Deploy postgresql-xl cluster

suo.li
Intro PostgreSQL PostgreSQL也叫Postgres,是一个开源的关系型数据库管理系统。其强调可扩展性和 对SQL语言的兼容性。其起源于加州大学伯克利分校的Ingres数据库,最先的名字叫 POSTGRES,为了突出对SQL的兼容性而改为了Postgr
Postgresql python

Postgresql python

Postgresql python

suo.li
Code #!/usr/bin/env python from psycopg2 import connect from random import randrange from sys import argv insertNu = argv[1] # create a connection conn = connect(database="testdb", user="postgres", password="", host="localhost", port="5432") print "Connection is OK." # data to insert nameList = ["Paul", "Allen", "Teddy", "Alice", "bob", "jack", "leslie", "cheung", "jay"] cityList = ["California", "Beijing", "Chendu", "Silicon Valley", "Lisbon", "Moby", "Heihe", "Neimeng", "Heilj"] cur = conn.cursor() for i in range(5001, int(insertNu)): cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES ("+ str(i + 1) +", \'"+ str(nameList[randrange(len(nameList))]) +"\', "+ str(randrange(20, 40)) +", \'"+ str(cityList[randrange(len(nameList))]) +"\', "+ str(float(randrange(20000,40000,3000))) +" )") conn.commit() print "Inserted." conn.close()
Mongodb shard cluster

Mongodb shard cluster

Mongodb shard cluster deploy

suo.li
Intro MongoDB shard(分片) 分片是MongoDB shard 集群中的一个概念,它将跨多个MongoDB实例的大型数据集拆分为 小型数据集存储在不同的分片(shard)中。 有时,MongoDB中的数据会非常庞大​​,以至于对如此大的数据集进行查询会导致服务器上
Mongodb install

Mongodb install

Mongodb install

suo.li
Stand alone installation install https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1804-4.2.7.tgz ~# pwd /usr/local/src ~# wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1804-4.2.7.tgz ~# tar -xf mongodb-linux-x86_64-ubuntu1804-4.2.7.tgz ~# ln -s /usr/local/mongodb-427 /usr/local/src/mongodb-linux-x86_64-ubuntu1804-4.2.7 ~# vim /etc/profile.d/mongodb.sh #!/usr/bin/env bash export PATH=/usr/local/mongodb-427/bin:$PATH ~# source /etc/profile.d/mongodb.sh edit mongod.conf ~# vim /etc/mongd.con processManagement: fork: true pidFilePath: /var/lib/mongo/db1.pid net: bindIp: localhost port: 27017 storage: dbPath: /var/lib/mongo systemLog: destination: file path: "/var/log/mongo/mongod.log" logAppend: true storage: journal: enabled: true replication: replSetName: csRS add mongodb user and edit service file add user mongodb ~# useradd -r 555 mongodb edit service file ~# vim /lib/systemd/system/mongodb.service [Unit] Description=High-performance, schema-free document-oriented database After=syslog.target network.target [Service] Type=forking User=mongodb Group=mongodb EnvironmentFile=/etc/mongod ExecStart=/usr/local/mongodb-427/bin/mongod $OPTIONS run PrivateTmp=true LimitNOFILE=64000 TimeoutStartSec=180 [Install] WantedBy=multi-user.target ~# vim /etc/mongd OPTIONS="--quiet -f /etc/mongod.conf" dirs and permissions ~# mkdir /var/{lib,log}/mongo -p ~# chown -R mongodb.mongodb /var/{lib,log}/mongo start mongodb ~# systemctl daemon-reload ~# systemctl start mongodb.
Redis高可用与集群

Redis高可用与集群

Redis HA and cluster

suo.li
Redis 高可用与集群 虽然 Redis 可以实现单机的数据持久化,但无论是 RDB 也好或者 AOF 也好,都解决 不了单点宕机问题,即一旦单台 redis 服务器本身出现系统故障、硬件故障等问题后, 就会直接造成数据的丢失,因此需要使用另外的技术来解决单点问题。 一.Redis 主从 主备模式,