$dev4life = "Software Development Blog"

MongoDB - Shard Distribution

One of the most important things in the MongoDB shard environment is to keep your data balanced among the shards.  The data might not be evenly distributed or not distributed at all for a number of reasons. A quick way to check your data distribution on a specific collection is to use getShardDistribution() command.  This commands provides you with a snapshot of the current state of each shard for a specific collection including the size of each shard and distribution percentage.  Shards could be unbalanced because of improper configuration, wrong shard key or because the balancer is not working properly or fast enough.  When a new shard is added, it could take days to re-balance all the shards depending on the data size.


Not Balanced Example

// MongoDB Shard Distribution
db.collection_name.getShardDistribution()


Shard shard0000 at d1:27018
 data : 80.85Gb docs : 93577613 chunks : 3106
 estimated data per chunk : 26.65Mb
 estimated docs per chunk : 30128

Shard shard0001 at d2:27018
 data : 79.72Gb docs : 92264345 chunks : 3106
 estimated data per chunk : 26.28Mb
 estimated docs per chunk : 29705

Shard shard0002 at d3:27018
 data : 73.64Gb docs : 85235367 chunks : 3106
 estimated data per chunk : 24.28Mb
 estimated docs per chunk : 27442

Shard shard0003 at d4:27018
 data : 70.99Gb docs : 82165553 chunks : 3107
 estimated data per chunk : 23.39Mb
 estimated docs per chunk : 26445

Shard shard0004 at d1:27020
 data : 8.36Gb docs : 9680754 chunks : 358
 estimated data per chunk : 23.92Mb
 estimated docs per chunk : 27041

Shard shard0005 at d2:27020
 data : 8.86Gb docs : 10265908 chunks : 358
 estimated data per chunk : 25.37Mb
 estimated docs per chunk : 28675

Shard shard0006 at d3:27020
 data : 7.91Gb docs : 9166138 chunks : 358
 estimated data per chunk : 22.65Mb
 estimated docs per chunk : 25603

Shard shard0007 at d4:27020
 data : 7.97Gb docs : 9234235 chunks : 357
 estimated data per chunk : 22.88Mb
 estimated docs per chunk : 25866

Totals
 data : 338.35Gb docs : 391589913 chunks : 13856
 Shard shard0000 contains 23.89% data, 23.89% docs in cluster, avg obj size on shard : 927b
 Shard shard0001 contains 23.56% data, 23.56% docs in cluster, avg obj size on shard : 927b
 Shard shard0002 contains 21.76% data, 21.76% docs in cluster, avg obj size on shard : 927b
 Shard shard0003 contains 20.98% data, 20.98% docs in cluster, avg obj size on shard : 927b
 Shard shard0004 contains 2.47% data, 2.47% docs in cluster, avg obj size on shard : 927b
 Shard shard0005 contains 2.62% data, 2.62% docs in cluster, avg obj size on shard : 927b
 Shard shard0006 contains 2.34% data, 2.34% docs in cluster, avg obj size on shard : 927b
 Shard shard0007 contains 2.35% data, 2.35% docs in cluster, avg obj size on shard : 927b


Balanced Example

// MongoDB Shard Distribution
db.collection_name.getShardDistribution()

Shard shard0000 at d1:27018
 data : 36.22GiB docs : 80083837 chunks : 1282
 estimated data per chunk : 28.93MiB
 estimated docs per chunk : 62467

Shard shard0001 at d2:27018
 data : 35.8GiB docs : 79154711 chunks : 1281
 estimated data per chunk : 28.61MiB
 estimated docs per chunk : 61791

Shard shard0002 at d3:27018
 data : 36.77GiB docs : 81312990 chunks : 1282
 estimated data per chunk : 29.37MiB
 estimated docs per chunk : 63426

Shard shard0003 at r2d4:27018
 data : 36.87GiB docs : 81519735 chunks : 1282
 estimated data per chunk : 29.45MiB
 estimated docs per chunk : 63587

Shard shard0004 at d1:27020
 data : 36.26GiB docs : 80174212 chunks : 1280
 estimated data per chunk : 29MiB
 estimated docs per chunk : 62636

Shard shard0005 at d2:27020
 data : 36.34GiB docs : 80358156 chunks : 1282
 estimated data per chunk : 29.03MiB
 estimated docs per chunk : 62681

Shard shard0006 at d3:27020
 data : 35.78GiB docs : 79113388 chunks : 1282
 estimated data per chunk : 28.58MiB
 estimated docs per chunk : 61710

Shard shard0007 at d4:27020
 data : 35.28GiB docs : 78020796 chunks : 1283
 estimated data per chunk : 28.16MiB
 estimated docs per chunk : 60811

Totals
 data : 289.35GiB docs : 639737825 chunks : 10254
 Shard shard0000 contains 12.51% data, 12.51% docs in cluster, avg obj size on shard : 485B
 Shard shard0001 contains 12.37% data, 12.37% docs in cluster, avg obj size on shard : 485B
 Shard shard0002 contains 12.71% data, 12.71% docs in cluster, avg obj size on shard : 485B
 Shard shard0003 contains 12.74% data, 12.74% docs in cluster, avg obj size on shard : 485B
 Shard shard0004 contains 12.53% data, 12.53% docs in cluster, avg obj size on shard : 485B
 Shard shard0005 contains 12.56% data, 12.56% docs in cluster, avg obj size on shard : 485B
 Shard shard0006 contains 12.36% data, 12.36% docs in cluster, avg obj size on shard : 485B
 Shard shard0007 contains 12.19% data, 12.19% docs in cluster, avg obj size on shard : 485B

database, getShardDistribution, mongodb, performance, sharding

SIP Response Codes (CSV, SQL, JSON)

SIP (Session Initiation Protocol) used for controlling VoIP (voice over IP), VoLTE (voice over LTE) and IMS signalling communication. Below you can download list of SIP codes in CSV, SQL and JSON formats to be used in an application. The list is generated from Wikipedia.





Downloads:
SIP Codes (CSV)
SIP Codes (SQL)
SIP Codes (JSON)


Sip Response Codes

100 - Trying
Extended search being performed may take a significant time so a forking proxy must send a 100 Trying response.

180 - Ringing
Destination user agent received INVITE, and is alerting user of call.

181 - Call is Being Forwarded
Servers can optionally send this response to indicate a call is being forwarded.

182 - Queued
Indicates that the destination was temporarily unavailable, so the server has queued the call until the destination is available. A server may send multiple 182 responses to update progress of the queue.

183 - Session in Progress
This response may be used to send extra information for a call which is still being set up.

199 - Early Dialog Terminated
Can be used by User Agent Server to indicate to upstream SIP entities (including the User Agent Client (UAC)) that an early dialog has been terminated.


200 - OK
Indicates the request was successful.

202 - Accepted
Indicates that the request has been accepted for processing, but the processing has not been completed.

204 - No Notification
Indicates the request was successful, but the corresponding response will not be received.


300 - Multiple Choices
The address resolved to one of several options for the user or client to choose between, which are listed in the message body or the message's Contact fields.

301 - Moved Permanently
The original Request-URI is no longer valid, the new address is given in the Contact header field, and the client should update any records of the original Request-URI with the new value.

302 - Moved Temporarily
The client should try at the address in the Contact field. If an Expires field is present, the client may cache the result for that period of time.

305 - Use Proxy
The Contact field details a proxy that must be used to access the requested destination.

380 - Alternative Service
The call failed, but alternatives are detailed in the message body.


400 - Bad Request
The request could not be understood due to malformed syntax.

401 - Unauthorized
The request requires user authentication. This response is issued by UASs and registrars.

402 - Payment Required
Reserved for future use.

403 - Forbidden
The server understood the request, but is refusing to fulfil it.

404 - Not Found
The server has definitive information that the user does not exist at the domain specified in the Request-URI. This status is also returned if the domain in the Request-URI does not match any of the domains handled by the recipient of the request.

405 - Method Not Allowed
The method specified in the Request-Line is understood, but not allowed for the address identified by the Request-URI.

406 - Not Acceptable
The resource identified by the request is only capable of generating response entities that have content characteristics but not acceptable according to the Accept header field sent in the request.

407 - Proxy Authentication Required
The request requires user authentication. This response is issued by proxys.

408 - Request Timeout
Couldn't find the user in time. The server could not produce a response within a suitable amount of time, for example, if it could not determine the location of the user in time. The client MAY repeat the request without modifications at any later time.

409 - Conflict
User already registered.

410 - Gone
The user existed once, but is not available here any more.

411 - Length Required
The server will not accept the request without a valid Content-Length.

412 - Conditional Request Failed
The given precondition has not been met.

413 - Request Entity Too Large
Request body too large.

414 - Request-URI Too Long
The server is refusing to service the request because the Request-URI is longer than the server is willing to interpret.

415 - Unsupported Media Type
Request body in a format not supported.

416 - Unsupported URI Scheme
Request-URI is unknown to the server.

417 - Unknown Resource-Priority
There was a resource-priority option tag, but no Resource-Priority header.

420 - Bad Extension
Bad SIP Protocol Extension used, not understood by the server.

421 - Extension Required
The server needs a specific extension not listed in the Supported header.

422 - Session Interval Too Small
The received request contains a Session-Expires header field with a duration below the minimum timer.

423 - Interval Too Brief
Expiration time of the resource is too short.

424 - Bad Location Information
The request's location content was malformed or otherwise unsatisfactory.

428 - Use Identity Header
The server policy requires an Identity header, and one has not been provided.

429 - Provide Referrer Identity
The server did not receive a valid Referred-By token on the request.

430 - Flow Failed
A specific flow to a user agent has failed, although other flows may succeed. This response is intended for use between proxy devices, and should not be seen by an endpoint (and if it is seen by one, should be treated as a 400 Bad Request response).

433 - Anonymity Disallowed
The request has been rejected because it was anonymous.

436 - Bad Identity-Info
The request has an Identity-Info header, and the URI scheme in that header cannot be dereferenced.

437 - Unsupported Certificate
The server was unable to validate a certificate for the domain that signed the request.

438 - Invalid Identity Header
The server obtained a valid certificate that the request claimed was used to sign the request, but was unable to verify that signature.

439 - First Hop Lacks Outbound Support
The first outbound proxy the user is attempting to register through does not support the 'outbound' feature of RFC 5626, although the registrar does.

470 - Consent Needed
The source of the request did not have the permission of the recipient to make such a request.

480 - Temporarily Unavailable
Callee currently unavailable.

481 - Call/Transaction Does Not Exist
Server received a request that does not match any dialog or transaction.

482 - Loop Detected
Server has detected a loop.

483 - Too Many Hops
Max-Forwards header has reached the value 0.

484 - Address Incomplete
Request-URI incomplete.

485 - Ambiguous
Request-URI is ambiguous.

486 - Busy Here
Callee is busy.

487 - Request Terminated
Request has terminated by bye or cancel.

488 - Not Acceptable Here
Some aspect of the session description or the Request-URI is not acceptable.

489 - Bad Event
The server did not understand an event package specified in an Event header field.

491 - Request Pending
Server has some pending request from the same dialog.

493 - Undecipherable
Request contains an encrypted MIME body, which recipient can not decrypt.

494 - Security Agreement Required
The server has received a request that requires a negotiated security mechanism, and the response contains a list of suitable security mechanisms for the requester to choose between.


500 - Server Internal Error
The server could not fulfill the request due to some unexpected condition.

501 - Not Implemented
The server does not have the ability to fulfill the request, such as because it does not recognize the request method. (Compare with 405 Method Not Allowed, where the server recognizes the method but does not allow or support it.)

502 - Bad Gateway
The server is acting as a gateway or proxy, and received an invalid response from a downstream server while attempting to fulfill the request.

503 - Service Unavailable
The server is undergoing maintenance or is temporarily overloaded and so cannot process the request. A 'Retry-After' header field may specify when the client may reattempt its request.

504 - Server Time-out
The server attempted to access another server in attempting to process the request, and did not receive a prompt response.

505 - Version Not Supported
The SIP protocol version in the request is not supported by the server.

513 - Message Too Large
The request message length is longer than the server can process.

580 - Precondition Failure
The server is unable or unwilling to meet some constraints specified in the offer.


600 - Busy Everywhere
All possible destinations are busy. Unlike the 486 response, this response indicates the destination knows there are no alternative destinations (such as a voicemail server) able to accept the call.

603 - Decline
The destination does not wish to participate in the call, or cannot do so, and additionally the destination knows there are no alternative destinations (such as a voicemail server) willing to accept the call.

604 - Does Not Exist Anywhere
The server has authoritative information that the requested user does not exist anywhere.

606 - Not Acceptable
The user's agent was contacted successfully but some aspects of the session description such as the requested media, bandwidth, or addressing style were not acceptable.

MySQL - Procedure Analyse

MySQL has a very useful built-in function that helps database administrators optimize the data structures and as a result, it can improve query performance and significantly decrease table size.

Procedure Analyse provides statistics on the data, such as maximum and minimum length of each data column as well as recommended optimal field type and length. In other words, if a field is set to a Signed INT (4 Bytes) and the maximum number that the field holds is no more than 255 and not less than 0, perhaps the field type should be changed to Unsigned TINYINT (1 Byte).
It also provides statistics on NULL values, empties or zeros. Using this information, database administrator can determine whether the field should be set to NOT NULL as well it may help pick the appropriate default value. Please refer to MySQL documentation for more details on data types.

This optimization is especially effective on large data sets since it could significantly decrease the size of the table and improve query performance.
Please note, in order to use PROCEDURE ANALYSE effectively, the table must be populated with some date.

Example:
SELECT * FROM transactions PROCEDURE ANALYSE(10,2000);

Procedure Analyse - database, monitoring, mysql, performance, tuning

MongoDB - Capped Collections

- Create collection
- View collection stats
- Convert existing collection to capped

One of the best features of MongoDB is ability to create capped collections.  Capped collection is a fixed size collection that deletes the old data while the new one is added.  This process is done seamlessly in the background.  Anyone familiar with MySQL and large data sets (over few Gigs) will know the frustration.  Administrator could simply set a collection to be of certain size (in bytes) and the database will take care of everything else.  There are however couple of things to note.  As of right now, capped collections can not be sharded.  Also, MongoDB pre-allocated space for capped collection.

Example
// Create collection
db.createCollection("normal_collection");

// Convert to capped collection (50MB capped)
db.normal_collection.convertToCapped(50000000);

// Check if converted to capped
db.normal_collection.isCapped();

// Create a new capped collection
db.createCollection("capped_collection", { capped: true, size: 50000000 });

// Check collection stats (shows size, count, index information, etc)
db.normal_collection.stats();

mongodb,capped collection, convertToCapped, createCollection, database, isCapped, stats

Linux Time Command - Measure Execution Time

Linux/Unix "time" Command

Time is a simple but very useful command to measure execution time.  It can be used for quickly benchmarking scripts and other programs. It provides three pieces of information, real is the actual time the program runs from start to end. User and Sys are time it took the cpu to run the code in user and system mode respectively.
Example
time php test.php
linux time command, bash, linux, performance, time command