MongoDB World 2014
by S.D.O.C. Ltd.Billing on top of MongoDBOfer Cohen
Who Am I ?
● Open source evangelist
● Former Board member of OpenSourceMatters
(Non-profit organization behind Joomla)
● S.D.O.C. Ltd. Co-Founder
FUD
● FUD: Fear, Uncertainty and Doubt
● Tactic used in sales, marketing, public
relations, politics and propaganda.
Wikipedia
FUD
Preferred name for the presentation
S.D.O.C. Ltd. vision
We increase business success through great
open source technologies and services○ Open & Transparent
○ Don't reinvent the wheel
○ High Quality
○ Lean & Effective
How did we get to billing (history)● Client: Golan Telecom (Israel)
How did we get to billing (history)
● Client: Golan Telecom ○ New & lean player in the market
○ 0=>~1M subscribers in few years
○ Limited resources & loves open source
How did we get to billing (history)● Client: Golan Telecom
○ Start up environment from Day 1
○ Short and aggressive time to market
○ Unlimited plan for ~25$
○ Customer can do almost everything using the website
■ Obviously requires 24/7 uptime
How did we get to billing (history)● Start with Anti-Fraud solution
● 2 Different data structure, 2 separated tables○ Outgoing calls (MOC)
○ incoming calls (MTC)
○ SMS (duration=0 means SMS)
Anti-Fraud in RDBMS How did it look like? Example code... $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start
. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start
$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";
if (isset($args['imsi']))
$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";
$base_query .= "GROUP BY imsi ";
$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration
"
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
Anti-Fraud in RDBMS How did it look like? Example code... . "FROM mtc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
$moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS
israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM moc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
Anti-Fraud in RDBMS How did it look like? Example code... $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "
. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "
. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "
. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "
. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS
moc_non_israel_duration_round, "
. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "
. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS
mtc_non_israel_duration_round, "
. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "
. "FROM "
. "( " . $base_query . " ) AS base "
. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "
. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;
Anti-Fraud in RDBMS How did it feel…?
After moving to Mongo
● One main collection for 2 types (MOC & MTC)
● Aggregation framework is much more simple
After moving to Mongo$base_match = array(
'$match' => array('source' => 'nrtrde','unified_record_time' => array('$gte' => new MongoDate($charge_time)),
));$where = array(
'$match' => array('record_type' => 'MOC','connectedNumber' => array('$regex' => '^972'),'event_stamp' => array('$exists' => false),'deposit_stamp' => array('$exists' => false),'callEventDurationRound' => array('$gt' => 0), // not sms
),);$group = array(
'$group' => array("_id" => '$imsi',"moc_israel" => array('$sum' => '$callEventDurationRound'),'lines_stamps' => array('$addToSet' => '$stamp'),
),);$project = array(
'$project' => array('imsi' => '$_id','_id' => 0,'moc_israel' => 1,'lines_stamps' => 1,
),);
After moving to Mongo$having = array(
'$match' => array(
'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))
),
);
$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);//sms out to all numbers
$where['$match']['record_type'] = 'MOC';
$where['$match']['callEventDurationRound'] = 0;
$group['$group']['sms_out'] = $group['$group']['mtc_all'];
unset($group['$group']['mtc_all']);
unset($having['$match']['mtc_all']);
$group['$group']['sms_out'] = array('$sum' => 1);
$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));
$project['$project']['sms_out'] = 1;
unset($project['$project']['mtc_all']);
$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);
What is billing?
● Group of processes of communications service providers
● responsible to collect consumption data● calculate charging and billing information● produce bills to customers● process their payments and manage debt
collectionWikipedia
What is billing?
● This is how we see it
● The KISS way
Telecom today - database challenges
● Telecom situation world wide today:
○ Unlimited packages, extend 3g usage, with high
competition
○ High-volume - 4g, LTE
○ Different Events - different data structure
5 *main* data-structures (CDR) from different sources
● NSN - Calls
● SMSC - SMS
● MMSC - MMS
● GGSN - Data
● TAP3 - International usage
Billing and MongoDB - Pros
NSN Record> db.lines.findOne({"usaget" : "call", aid:XXXXXXX})
{
"_id" : ObjectId("52bafd818f7ac3943a8b96dc"),
"stamp" : "87cea5dec484c8f6a19e44e77a2e15b4",
"record_type" : "01",
"record_number" : "13857000",
"record_status" : 0,
"exchange_id" : "000006270000",
"call_reference" : "700227d9a3",
"urt" : ISODate("2013-12-25T15:09:15Z"),
"charging_start_time" : "20131225170915",
"charging_end_time" : "20131225171517",
"call_reference_time" : "20131225170914",
"duration" : 362,
"calling_number" : "972546918666",
"called_number" : "26366667",
"call_type" : "3",
"chrg_type" : "0",
"called_imsi" : "000030000000000",
"imsi" : "000089000101076",
"in_circuit_group_name" : "",
"in_circuit_group" : "",
"out_circuit_group_name" : "NBZQZA8",
"out_circuit_group" : "0503",
"tariff_class" : "000000",
"called_number_ton" : "6",
"org_dur" : 362,
"type" : "nsn",
"source" : "binary",
"file" : "CF0322.DAT",
"log_stamp" : "0a3ffdb7c9ccc175e38be2fcc00f8c28",
"process_time" : "2013-12-25 17:35:22",
"usaget" : "call","usagev" : 362,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001c9")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
}
SMS Record> db.lines.findOne({"usaget" : "sms", aid:XXXXXXX})
{
"_id" : ObjectId("52e52ff1d88db071648b4ad7"),
"stamp" : "9328f3aaa114aaba910910053a11b3e8",
"record_type" : "1",
"calling_number" : "000972546918666",
"calling_imsi" : "000089200000000",
"calling_msc" : "000972000000000",
"billable" : "000000000000000",
"called_number" : "000972547655380",
"called_imsi" : "425089109386379",
"called_msc" : "000972586279101",
"message_submition_time" : "140125192517",
"time_offest" : "02",
"message_delivery_time" : "140125192519",
"time_offest1" : "02",
"cause_of_terminition" : "100",
"call_reference" : "5137864939035049",
"message_length" : "050",
"concatenated" : "1",
"concatenated_from" : "09",
"source" : "separator_field_lines",
"type" : "smsc",
"log_stamp" : "a5950686e364d1400c13dd1857c3340e",
"file" : "140125192403_5735golan.cdr",
"process_time" : "2014-01-26 17:53:21",
"urt" : ISODate("2014-01-25T17:25:17Z"),
"usaget" : "sms","usagev" : 1,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001db")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 4,
"billrun" : "201402"
}
Data Record> db.lines.findOne({"usaget" : "data", aid:XXXXXXX})
{
"_id" : ObjectId("539076678f7ac34a1d8b9367"),
"stamp" : "8a9f891ec85c5294c974a34653356055",
"imsi" : "400009209100000",
"served_imsi" : "400009209100000",
"ggsn_address" : "XX.XX.144.18",
"charging_id" : "2814645234",
"sgsn_address" : "XX.XX.145.9",
"served_pdp_address" : "XX.XX.237.95",
"urt" : ISODate("2014-06-05T09:34:47Z"),
"record_opening_time" : "20140605123447",
"ms_timezone" : "+03:00",
"node_id" : "GLTNGPT",
"served_msisdn" : "00002546918666",
"fbc_uplink_volume" : 61298,
"fbc_downlink_volume" : 217304,
"rating_group" : 0,
"type" : "ggsn",
"source" : "binary",
"file" : "GLTNGPT_-_0000056580.20140605_-_1251+0300",
"log_stamp" : "45a227ced1098bc76a44774eae04eb67",
"process_time" : "2014-06-05 16:39:40",
"usaget" : "data","usagev" : 278602,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f000200")),
"apr" : 0.020264916503503202,
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 478682116,
"billrun" : "201406"
}
TAP3 Record (intl roaming)> db.lines.findOne({type:"tap3", aid:9073496}){
"_id" : ObjectId("538d9ac98f7ac3e17d8b4fd6"),"stamp" : "8f6cdc8662307ee2ed951ce640a585b5","basicCallInformation" : {
"GprsChargeableSubscriber" : {"chargeableSubscriber" : {
"simChargeableSubscriber" : {"imsi" : "400009209100000"
}},"pdpAddress" : "XX.XX.227.158"
},"GprsDestination" : {
"AccessPointNameNI" : "internet.golantelecom.net.il"},"CallEventStartTimeStamp" : {
"localTimeStamp" : "20140529205131","TimeOffsetCode" : 0
},"TotalCallEventDuration" : 163
},"LocationInformation" : {
"gprsNetworkLocation" : {"RecEntityCodeList" : {
"RecEntityCode" : ["","\u0000"
]},"LocationArea" : 00001,"CellId" : 0001
},"GeographicalLocation" : {
"ServingNetwork" : "MMMM"}
},"ImeiOrEsn" : false,
"GprsServiceUsed" : {"DataVolumeIncoming" : 195120,"DataVolumeOutgoing" : 48600,"ChargeInformationList" : {
"ChargeInformation" : { "ChargedItem" : "X", "ExchangeRateCode" : 0, "ChargeDetailList" : { "ChargeDetail" : {
"ChargeType" : "00", "Charge" : 001, "ChargeableUnits" : 100000, "ChargedUnits" : 100000, "ChargeDetailTimeStamp" : { "localTimeStamp" : "20140529205131", "TimeOffset" : 0 } }
}}
}},"OperatorSpecInfoList" : {
"OperatorSpecInformation" : ["00000000.0000","00000000.0000","00000000.0000"
]},"record_type" : "e","urt" : ISODate("2014-05-29T18:51:31Z"),"tzoffset" : "+0200","imsi" : "400009209100000","serving_network" : "DEUE2","sdr" : 0.0001,"exchange_rate" : 1.12078,"type" : "tap3","file" : "CDBELHBISRGT02253",
"log_stamp" : "a0ad109c6e795f6c1feeef9ef649d937","process_time" : "2014-06-03 12:50:08",
"usaget" : "data","usagev" : 243720,"arate" : DBRef("rates", ObjectId
("521e07fed88db0e73f000219")),"apr" : 0.46640616,"aid" : 9073496,"sid" : 78288,"plan" : "LARGE","out_plan" : 243720,"aprice" : 0.46640616,"usagesb" : 39139746,"billrun" : "201406"
}
Billing and MongoDB - ProsLoose coupling compared to traditional billing components
Oldw/o MongoDb New
with MongoDb
Billing and MongoDB - Pros● One call can be spread on few CDRs● BillRun merges records only on presentation
layer or export● No DB-level aggregation nor accumulation
○ No need for mediation● Use billing and monitoring CDR in one system
database
Billing and MongoDB - Pros
Sophisticated rating module
● Rating module depends on CDR structure
● Easy to implement recurring rating
Rate module example
MongoDB advantages with Billing
Invoice JSON2PDF process● Use json as metadata for the PDF
● Easy to export
● Fast processing
MongoDB advantages with BillingInvoice metadata
> db.billrun.findOne({billrun_key:"201405", aid:9073496}){ "aid" : NumberLong(9073496), "subs" : [ { "sid" : NumberLong(78288), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "kosher" : false, "breakdown" : { "in_plan" : { "base" : { "INTERNET_BILL_BY_VOLUME" : { "totals" : { "data" : { "usagev" : NumberLong(1975547725), "cost" : NumberLong(0), "count" : NumberLong(1352) } }, "vat" : 0.18 }, "IL_MOBILE" : { "totals" : { "sms" : { "usagev" : NumberLong(159), "cost" : NumberLong(0), "count" : NumberLong(159) }, "call" : { "usagev" : NumberLong(20788), "cost" : NumberLong(0), "count" : NumberLong(83) } }, "vat" : 0.18 },
"IL_FIX" : { "totals" : { "call" : { "usagev" : NumberLong(1217), "cost" : NumberLong(0), "count" : NumberLong(16) } }, "vat" : 0.18 }, "INTERNAL_VOICE_MAIL_CALL" : { "totals" : { "call" : { "usagev" : NumberLong(60), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 }, "service" : { "cost" : 83.898305085, "vat" : 0.18 } }, "intl" : { "KT_USA_NEW" : { "totals" : { "call" : { "usagev" : NumberLong(149), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 } } },
MongoDB advantages with BillingInvoice metadata
"credit" : { "refund_vatable" : { "CRM-REFUND_PROMOTION_1024-BILLRUN_201405" : -33.898305084746 } } }, "lines" : { "data" : { "counters" : { "20140425" : { "usagev" : NumberLong(11991615), "aprice" : NumberLong(0), "plan_flag" : "in" }, …. /* data by date really long, so let’s cut it from this demonstration */ } } }, "totals" : { "vatable" : 50.000000000254005, "before_vat" : 50.000000000254005, "after_vat" : 59.00000000029973 }, "costs" : { "credit" : { "refund" : { "vatable" : -33.898305084746 } }, "flat" : { "vatable" : 83.898305085 } } }, { "sid" : NumberLong(354961), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")),
"kosher" : false, "breakdown" : { "in_plan" : { "base" : { "service" : { "cost" : 8.466101695, "vat" : 0.18 } } } }, "totals" : { "vatable" : 8.466101695, "before_vat" : 8.466101695, "after_vat" : 9.9900000001 }, "costs" : { "flat" : { "vatable" : 8.466101695 } } } ], "vat" : 0.18, "billrun_key" : "201405", "totals" : { "before_vat" : 58.466101695254004, "after_vat" : 68.99000000039973, "vatable" : 58.466101695254004 }, "_id" : ObjectId("5382fd3cd88db0c31a8b74cc"), "invoice_id" : NumberLong(14738102), "invoice_file" : "201405_009073496_00014738102.xml"}
Infrastructure
BETAFirst Production
Today
Data center 1
Data center 2
Data center 1
Data center 2 Data center 1
Data center 2
Archive
App stack
BillRun application
BillRun core
PHP YAF framework
Zend and more libraries
MongoDB
Web service Cli/Cron services
Mongodloid https://github.com/BillRun/Mongodloid
https://github.com/mongodb/mongo/
https://github.com/BillRun/system
https://github.com/zendframework/https://github.com/twbs/bootstraphttps://github.com/laruence/php-yaf
Pay Attention! What to keep in mind
Transactional (tx) processes● write concern - acknowledged (default in 2.4)● findAndModify (A.K.A FAM) - document tx● value=oldValue● 2 phase commit - app side● Transaction lock by design
Transaction workflow diagram
High performance tricks● With SSD you get x20 more performance● MongoDB loves RAM● Follow MongoDB production notes
○ Readahead low as possible○ THP - transparent hugepages
Pay Attention! What to keep in mind
More tips that happened to all● Shorten property names● MongoDB have database lock
○ Separate database for heavy-write collections
Pay Attention! What to keep in mind
TCO - MongoDB based solution
● 3 Months dev
● 3 Months QA
● Few days of maintenance infra and app
● Easy to add features
● Easy to scale
What can BillRun do
● 5,000 events per second○ Including DB insert and transactional rating
● That means 157,680,000,000 events per year
● Create hundreds of millions invoices
Do not forget!
Top Related