Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 32 What syslog-ng...
Transcript of Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 32 What syslog-ng...
syslog-ngOpenSourceEdition3.18
AdministrationGuide
Copyright 2018 One Identity LLC.
ALL RIGHTS RESERVED.
Thisguidecontainsproprietaryinformationprotectedbycopyright.Thesoftwaredescribedinthisguideisfurnishedunderasoftwarelicenseornondisclosureagreement.Thissoftwaremaybeusedorcopiedonlyinaccordancewiththetermsoftheapplicableagreement.Nopartofthisguidemaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,includingphotocopyingandrecordingforanypurposeotherthanthepurchaserspersonalusewithoutthewrittenpermissionofOneIdentityLLC.TheinformationinthisdocumentisprovidedinconnectionwithOneIdentityproducts.Nolicense,expressorimplied,byestoppelorotherwise,toanyintellectualpropertyrightisgrantedbythisdocumentorinconnectionwiththesaleofOneIdentityLLCproducts.EXCEPTASSETFORTHINTHETERMSANDCONDITIONSASSPECIFIEDINTHELICENSEAGREEMENTFORTHISPRODUCT,ONEIDENTITYASSUMESNOLIABILITYWHATSOEVERANDDISCLAIMSANYEXPRESS,IMPLIEDORSTATUTORYWARRANTYRELATINGTOITSPRODUCTSINCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTYOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ORNON-INFRINGEMENT.INNOEVENTSHALLONEIDENTITYBELIABLEFORANYDIRECT,INDIRECT,CONSEQUENTIAL,PUNITIVE,SPECIALORINCIDENTALDAMAGES(INCLUDING,WITHOUTLIMITATION,DAMAGESFORLOSSOFPROFITS,BUSINESSINTERRUPTIONORLOSSOFINFORMATION)ARISINGOUTOFTHEUSEORINABILITYTOUSETHISDOCUMENT,EVENIFONEIDENTITYHASBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.OneIdentitymakesnorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisdocumentandreservestherighttomakechangestospecificationsandproductdescriptionsatanytimewithoutnotice.OneIdentitydoesnotmakeanycommitmenttoupdatetheinformationcontainedinthisdocument.Ifyouhaveanyquestionsregardingyourpotentialuseofthismaterial,contact:
OneIdentityLLC.Attn:LEGALDept4PolarisWayAlisoViejo,CA92656
RefertoourWebsite(http://www.OneIdentity.com)forregionalandinternationalofficeinformation.
Patents
OneIdentityisproudofouradvancedtechnology.Patentsandpendingpatentsmayapplytothisproduct.Forthemostcurrentinformationaboutapplicablepatentsforthisproduct,pleasevisitourwebsiteathttp://www.OneIdentity.com/legal/patents.aspx.
Trademarks
OneIdentityandtheOneIdentitylogoaretrademarksandregisteredtrademarksofOneIdentityLLC.intheU.S.A.andothercountries.ForacompletelistofOneIdentitytrademarks,pleasevisitourwebsiteatwww.OneIdentity.com/legal.Allothertrademarksarethepropertyoftheirrespectiveowners.
Legend
WARNING: A WARNING icon indicates a potential for property damage, personal injury, or death.
CAUTION: A CAUTION icon indicates potential damage to hardware or loss of data if instructions are not followed.
IMPORTANT,NOTE,TIP,MOBILE,orVIDEO:Aninformationiconindicatessupportinginformation.
syslog-ngOSEAdministrationGuideUpdated-November2018Version-3.18
http://www.oneidentity.com/http://www.oneidentity.com/legal/patents.aspxhttp://www.oneidentity.com/legal
Contents
Preface 19
Summaryofcontents 19
Targetaudienceandprerequisites 20
Productscoveredinthisguide 20
Summaryofchanges 21
Version3.17-3.18 21
Version3.16-3.17 22
Version3.15-3.16 22
Version3.14-3.15 23
Version3.13-3.14 23
Version3.12-3.13 24
Version3.11-3.12 25
Version3.10-3.11 26
Version3.9-3.10 26
Version3.8-3.9 27
Version3.7-3.8 28
Version3.6-3.7 29
Version3.5-3.6 30
Acknowledgments 31
Introduction to syslog-ng 32
Whatsyslog-ngis 32
Secureandreliablelogtransfer 32
Flexibledataextractionandprocessing 33
Bigdataclusters 33
Messagequeuesupport 33
SQL,NoSQL,andmonitoring 34
Wideprotocolandplatformsupport 34
Whatsyslog-ngisnot 34
Whyissyslog-ngneeded? 34
Whatisnewinsyslog-ngOpenSourceEdition3.18? 35
Whousessyslog-ng? 36
syslog-ng OSE 3.18 Administration Guide 3
Supportedplatforms 36
The concepts of syslog-ng 38
Thephilosophyofsyslog-ng 38
Loggingwithsyslog-ng 38
Therouteofalogmessageinsyslog-ng 39
Modesofoperation 40
Clientmode 40
Relaymode 41
Servermode 41
Globalobjects 42
Timezonesanddaylightsaving 43
Howsyslog-ngOSEassignstimezonetothemessage 44
Anoteontimezonesandtimestamps 45
Productlicensing 45
Highavailabilitysupport 45
Thestructureofalogmessage 45
BSD-syslogorlegacy-syslogmessages 46
ThePRImessagepart 46
TheHEADERmessagepart 48
TheMSGmessagepart 48
IETF-syslogmessages 48
ThePRImessagepart 49
TheHEADERmessagepart 50
TheSTRUCTURED-DATAmessagepart 51
TheMSGmessagepart 51
Enterprise-widemessagemodel(EWMM) 51
Messagerepresentationinsyslog-ngOSE 52
Structuringmacros,metadata,andothervalue-pairs 54
Specifyingdatatypesinvalue-pairs 55
value-pairs() 56
Thingstoconsiderwhenforwardingmessagesbetweensyslog-ngOSEhosts 61
Commercialversionofsyslog-ng 63
Installing syslog-ng 65
Compilingsyslog-ngfromsource 65
syslog-ng OSE 3.18 Administration Guide 4
Compilingoptionsofsyslog-ngOSE 67
Uninstallingsyslog-ngOSE 70
ConfiguringMicrosoftSQLServertoacceptlogsfromsyslog-ng 70
The syslog-ng OSE quick-start guide 77
Configuringsyslog-ngonclienthosts 77
Configuringsyslog-ngonserverhosts 80
Configuringsyslog-ngrelays 82
Configuringsyslog-ngonrelayhosts 82
Howrelayinglogmessagesworks 84
The syslog-ng OSE configuration file 86
Locationofthesyslog-ngconfigurationfile 86
Theconfigurationsyntaxindetail 86
Notesabouttheconfigurationsyntax 89
Definingconfigurationobjectsinline 90
Usingchannelsinconfigurationobjects 91
Globalandenvironmentalvariables 93
Modulesinsyslog-ngOSE 94
Loadingmodules 94
Managingcomplexsyslog-ngconfigurations 95
Includingconfigurationfiles 95
Reusingconfigurationblocks 96
Mandatoryparameters 98
Passingargumentstoconfigurationblocks 99
Generatingconfigurationblocksfromascript 100
Pythoncodeinexternalfiles 102
source: Read, receive, and collect log messages 104
Howsourceswork 105
default-network-drivers:Receiveandparsecommonsyslogmessages 108
default-network-drivers()sourceoptions 110
internal:Collectinginternalmessages 113
internal()sourceoptions 114
file:Collectingmessagesfromtextfiles 115
Notesonreadingkernelmessages 116
file()sourceoptions 116
syslog-ng OSE 3.18 Administration Guide 5
wildcard-file:Collectingmessagesfrommultipletextfiles 127
wildcard-file()sourceoptions 128
linux-audit:CollectingmessagesfromLinuxauditlogs 141
linux-audit()sourceoptions 142
network:CollectingmessagesusingtheRFC3164protocol(network()driver) 143
network()sourceoptions 144
nodejs:ReceivingJSONmessagesfromnodejsapplications 156
nodejs()sourceoptions 157
mbox:Convertinglocale-mailmessagestologmessages 159
mbox()sourceoptions 160
osquery:Collectandparseosqueryresultlogs 161
osquery()sourceoptions 164
pipe:Collectingmessagesfromnamedpipes 167
pipe()sourceoptions 167
pacct:CollectingprocessaccountinglogsonLinux 178
pacct()options 178
program:Receivingmessagesfromexternalapplications 180
program()sourceoptions 181
python:writingserver-stylePythonsources 187
Methodsofthepython()source 190
PythonLogMessageAPI 191
python()andpython-fetcher()sourceoptions 192
python-fetcher:writingfetcher-stylePythonsources 197
Methodsofthepython-fetcher()source 199
snmptrap:ReadNet-SNMPtraps 201
snmptrap()sourceoptions 204
sun-streams:CollectingmessagesonSunSolaris 207
sun-streams()sourceoptions 208
syslog:CollectingmessagesusingtheIETFsyslogprotocol(syslog()driver) 214
syslog()sourceoptions 216
system:Collectingthesystem-specificlogmessagesofaplatform 227
system()sourceoptions 229
systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstorage 232
systemd-journal()sourceoptions 234
systemd-syslog:Collectingsystemdmessagesusingasocket 238
syslog-ng OSE 3.18 Administration Guide 6
systemd-syslog()sourceoptions 239
tcp,tcp6,udp,udp6:CollectingmessagesfromremotehostsusingtheBSDsyslogprotocolOBSOLETE 241
tcp(),tcp6(),udp()andudp6()sourceoptions:OBSOLETE 241
Changeanoldsourcedrivertothenetwork()driver 242
unix-stream,unix-dgram:CollectingmessagesfromUNIXdomainsockets 243
UNIXcredentialsandothermetadata 243
unix-stream()andunix-dgram()sourceoptions 244
stdin:Collectingmessagesfromthestandardinputstream 253
stdin()sourceoptions 253
destination: Forward, send, and store log messages 264
amqp:PublishingmessagesusingAMQP 266
amqp()destinationoptions 267
elasticsearch:SendingmessagesdirectlytoElasticsearchversion1.x(DEPRECATED) 278
Prerequisites 280
Howsyslog-ngOSEinteractswithElasticsearch 281
Clientmodes 282
Elasticsearchdestinationoptions 282
elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher 294
Prerequisites 297
Howsyslog-ngOSEinteractswithElasticsearch 297
Clientmodes 298
SearchGuardandsyslog-ngOSE 299
Elasticsearch2destinationoptions 301
ExampleusecasesofsendinglogstoElasticsearchusingsyslog-ng 321
file:Storingmessagesinplain-textfiles 321
file()destinationoptions 322
graphite:SendingmetricstoGraphite 333
graphite()destinationoptions 334
SendinglogstoGraylog 337
graylog2()destinationoptions 338
hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS) 340
Prerequisites 341
Howsyslog-ngOSEinteractswithHDFS 342
syslog-ng OSE 3.18 Administration Guide 7
StoringmessageswithMapR-FS 343
Kerberosauthenticationwithsyslog-nghdfs()destination 344
HDFSdestinationoptions 345
PostingmessagesoverHTTP 356
HTTPdestinationoptions 357
http:PostingmessagesoverHTTPwithoutJava 361
Batchmode 363
Batchsize 363
Formattingthebatch 363
HTTPdestinationoptions 364
kafka:PublishingmessagestoApacheKafka 379
Prerequisites 381
Howsyslog-ngOSEinteractswithApacheKafka 382
Kafkadestinationoptions 382
loggly:UsingLoggly 388
loggly()destinationoptions 390
logmatic:UsingLogmatic.io 391
logmatic()destinationoptions 393
mongodb:StoringmessagesinaMongoDBdatabase 395
Howsyslog-ngOSEconnectstheMongoDBserver 396
mongodb()destinationoptions 397
network:SendingmessagestoaremotelogserverusingtheRFC3164protocol(network()driver) 406
network()destinationoptions 407
osquery:Sendinglogmessagestoosquery'ssyslogtable 423
osquery()destinationoptions 424
pipe:Sendingmessagestonamedpipes 426
pipe()destinationoptions 427
program:Sendingmessagestoexternalapplications 433
program()destinationoptions 435
pseudofile() 444
pseudofile()destinationoptions 444
python:writingcustomPythondestinations 446
Methodsofthepython()destination 448
Errorhandlinginthepython()destination 449
syslog-ng OSE 3.18 Administration Guide 8
python()destinationoptions 451
redis:Storingname-valuepairsinRedis 457
redis()destinationoptions 458
riemann:MonitoringyourdatawithRiemann 464
riemann()destinationoptions 465
smtp:GeneratingSMTPmessages(e-mail)fromlogs 476
smtp()destinationoptions 478
Splunk:SendinglogmessagestoSplunk 486
sql:StoringmessagesinanSQLdatabase 486
Usingthesql()driverwithanOracledatabase 488
Usingthesql()driverwithaMicrosoftSQLdatabase 489
Thewaysyslog-nginteractswiththedatabase 491
MySQL-specificinteractionmethods 492
MsSQL-specificinteractionmethods 492
sql()destinationoptions 492
stomp:PublishingmessagesusingSTOMP 504
stomp()destinationoptions 505
syslog:SendingmessagestoaremotelogserverusingtheIETF-syslogprotocol 511
syslog()destinationoptions 512
syslog-ng():Forwardlogstoanothersyslog-ngnode 528
syslog-ng()destinationoptions 529
tcp,tcp6,udp,udp6:SendingmessagestoaremotelogserverusingthelegacyBSD-syslogprotocol(tcp(),udp()drivers) 542
tcp(),tcp6(),udp(),andudp6()destinationoptions 543
Changeanolddestinationdrivertothenetwork()driver 543
Telegram:SendingmessagestoTelegram 544
telegram()destinationoptions 545
unix-stream,unix-dgram:SendingmessagestoUNIXdomainsockets 548
unix-stream()andunix-dgram()destinationoptions 548
usertty:Sendingmessagestoauserterminal:usertty()destination 558
WriteyourowncustomdestinationinJavaorPython 558
Client-sidefailover 558
log: Filter and route log messages using log paths, flags, and filters 561
Logpaths 561
Embeddedlogstatements 562
syslog-ng OSE 3.18 Administration Guide 9
Usingembeddedlogstatements 564
if-else-elif:Conditionalexpressions 566
Junctionsandchannels 566
Logpathflags 569
Managingincomingandoutgoingmessageswithflow-control 572
Flow-controlandmultipledestinations 576
Configuringflow-control 576
Usingdisk-basedandmemorybuffering 578
Enablingreliabledisk-basedbuffering 580
Enablingnormaldisk-basedbuffering 581
Enablingmemorybuffering 581
Aboutdiskqueuefiles 582
Filters 583
Usingfilters 583
Combiningfilterswithbooleanoperators 584
Comparingmacrovaluesinfilters 585
Usingwildcards,specialcharacters,andregularexpressionsinfilters 586
Taggingmessages 587
Filterfunctions 588
Droppingmessages 593
Global options of syslog-ng OSE 595
Configuringglobalsyslog-ngoptions 595
Globaloptions 595
TLS-encrypted message transfer 613
SecureloggingusingTLS 613
EncryptinglogmessageswithTLS 614
ConfiguringTLSonthesyslog-ngclients 615
ConfiguringTLSonthesyslog-ngserver 616
MutualauthenticationusingTLS 618
ConfiguringTLSonthesyslog-ngclients 619
ConfiguringTLSonthesyslog-ngserver 620
Password-protectedkeys 622
TLSoptions 623
template and rewrite: Format, modify, and manipulate log messages 630
syslog-ng OSE 3.18 Administration Guide 10
Customizemessageformatusingmacrosandtemplates 630
Formattingmessages,filenames,directories,andtablenames 631
Templatesandmacros 631
Date-relatedmacros 633
Hardvs.softmacros 634
Macrosofsyslog-ngOSE 635
Usingtemplatefunctions 644
Templatefunctionsofsyslog-ngOSE 645
Modifyingtheon-the-wiremessageformat 668
Modifyingmessagesusingrewriterules 669
Replacingmessageparts 669
Settingmessagefieldstospecificvalues 671
Unsettingmessagefields 674
CreatingcustomSDATAfields 675
Settingmultiplemessagefieldstospecificvalues 676
map-value-pairs:Renamevalue-pairstonormalizelogs 677
Conditionalrewrites 677
Howconditionalrewritingworks 678
Addinganddeletingtags 678
Anonymizingcreditcardnumbers 679
Regularexpressions 680
Typesandoptionsofregularexpressions 681
Optimizingregularexpressions 682
parser: Parse and segment structured messages 684
Parsingsyslogmessages 685
Optionsofsyslog-parserparsers 687
Parsingmessageswithcomma-separatedandsimilarvalues 689
OptionsofCSVparsers 692
Parsingkey=valuepairs 696
Optionsofkey=valueparsers 699
The JSON parser 700
TheJSONparserTheJSONparser 700
OptionsofJSONparsers 703
TheXMLparser 705
syslog-ng OSE 3.18 Administration Guide 11
OptionsofXMLparsers 709
Parsingdatesandtimestamps 711
Optionsofdate-parser()parsers 712
TheApacheAccessLogParser 714
Optionsofapache-accesslog-parser()parsers 715
TheCiscoParser 716
TheLinuxAuditParser 718
Optionsoflinux-audit-parser()parsers 720
ThePythonParser 721
Parsingenterprise-widemessagemodel(EWMM)messages 727
Thesudoparser 727
Theiptablesparser 728
db-parser: Process message content with a pattern database (patterndb) 730
Classifyinglogmessages 730
Thestructureofthepatterndatabase 731
Howpatternmatchingworks 732
Artificialignorance 733
Usingpatterndatabases 734
Usingparserresultsinfiltersandtemplates 735
Downloadingsamplepatterndatabases 737
Correlatinglogmessagesusingpatterndatabases 738
Referencingearliermessagesofthecontext 740
Triggeringactionsforidentifiedmessages 741
Conditionalactions 743
Externalactions 744
Actionsandmessagecorrelation 745
Creatingpatterndatabases 748
Usingpatternparsers 748
Patternparsersofsyslog-ngOSE 750
What'snewinthesyslog-ngpatterndatabaseformatV5 753
Thesyslog-ngpatterndatabaseformat 753
Element:patterndb 755
Element:ruleset 755
Element:patterns 756
Element:rules 757
syslog-ng OSE 3.18 Administration Guide 12
Element:rule 758
Element:patterns 760
Element:urls 761
Element:values 762
Element:examples 762
Element:example 763
Element:actions 764
Element:action 766
Element:create-context 768
Element:tags 771
Correlating log messages 772
Correlatingmessagesusingthegrouping-by()parser 772
Referencingearliermessagesofthecontext 776
Optionsofgrouping-byparsers 777
Enriching log messages with external data 781
Addingmetadatafromanexternalfile 781
Usingfiltersasselector 783
Optionsadd-contextual-data() 784
LookingupGeoIPdatafromIPaddresses(DEPRECATED) 786
Optionsofgeoipparsers 788
LookingupGeoIP2datafromIPaddresses 789
Referringtopartsofthemessageasamacro 790
UsingtheGeoIP2parser 790
TransferringyourlogstoElasticsearchusingGeoIP2 791
Optionsofgeoip2parsers 792
Statistics of syslog-ng 794
Metricsandcountersofsyslog-ngOSE 794
Logstatisticsfromtheinternal()source 797
Multithreading and scaling in syslog-ng OSE 799
Multithreadingconceptsofsyslog-ngOSE 799
Configuringmultithreading 801
Optimizingmultithreadedperformance 801
Troubleshooting syslog-ng 803
syslog-ng OSE 3.18 Administration Guide 13
Possiblecausesoflosinglogmessages 804
Creatingsyslog-ngcorefiles 805
Collectingdebugginginformationwithstrace,truss,ortusc 805
Runningafailurescript 806
Stoppingsyslog-ng 807
Reportingbugsandfindinghelp 808
Recoverdatafromorphaneddiskbufferfiles 808
Nolocallogsafterspecifyinganunusualstoragedirectory 808
Nologsafterspecifyinganunusualportnumber 808
Errormessages 809
Best practices and examples 811
Generalrecommendations 811
Handlinglargemessageload 811
Usingnameresolutioninsyslog-ng 812
Resolvinghostnameslocally 813
Collectinglogsfromchroot 813
Configuringlogrotation 814
The syslog-ng manual pages 816
Thedqtooltoolmanualpage 816
Name 816
Synopsis 816
Description 816
Thecatcommand 817
Files 818
Seealso 818
Author 818
Copyright 818
Theloggenmanualpage 818
Name 819
Synopsis 819
Description 819
Options 819
Examples 822
Files 822
syslog-ng OSE 3.18 Administration Guide 14
Seealso 822
Author 823
Copyright 823
Thepdbtoolmanualpage 823
Name 823
Synopsis 823
Description 824
Thedictionarycommand 824
Thedumpcommand 824
Thematchcommand 825
Themergecommand 827
Thepatternizecommand 828
Thetestcommand 829
Files 829
Seealso 829
Author 830
Copyright 830
Thesyslog-ngcontroltoolmanualpage 830
Name 830
Synopsis 830
Description 831
Enablingtroubleshootingmessages 831
syslog-ng-ctlquery 832
Thestatscommand 834
Handlingpassword-protectedprivatekeys 835
Displayingtheconfiguration 836
Reloadingtheconfiguration 836
Files 837
Seealso 837
Author 837
Copyright 837
Thesyslog-ng-debunmanualpage 837
Name 838
Synopsis 838
Description 838
syslog-ng OSE 3.18 Administration Guide 15
GeneralOptions 838
Debugmodeoptions 839
Systemcalltracing 839
Packetcaptureoptions 839
Examples 840
Files 841
Seealso 841
Author 841
Copyright 841
Thesyslog-ngmanualpage 841
Name 842
Synopsis 842
Description 842
Options 842
Files 845
Seealso 845
Author 845
Copyright 845
Thesyslog-ng.confmanualpage 846
Name 846
Synopsis 846
Description 846
Basicconceptsofsyslog-ngOSE 846
Configuringsyslog-ng 847
Files 851
Seealso 852
Author 852
Copyright 852
Third-party contributions 853
GNUGeneralPublicLicense 853
Preamble 853
TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 854
Section0 854
Section1 855
Section2 855
syslog-ng OSE 3.18 Administration Guide 16
Section3 856
Section4 856
Section5 856
Section6 857
Section7 857
Section8 857
Section9 858
Section10 858
NOWARRANTYSection11 858
Section12 858
HowtoApplyTheseTermstoYourNewPrograms 859
GNULesserGeneralPublicLicense 860
Preamble 860
TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 862
Section0 862
Section1 862
Section2 863
Section3 863
Section4 864
Section5 864
Section6 865
Section7 866
Section8 866
Section9 866
Section10 866
Section11 867
Section12 867
Section13 867
Section14 868
NOWARRANTYSection15 868
NOWARRANTYSection16 868
HowtoApplyTheseTermstoYourNewLibraries 868
Licenseattributions 869
Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License 870
syslog-ng OSE 3.18 Administration Guide 17
About us 876
Contactingus 876
Technicalsupportresources 876
Glossary 877
syslog-ng OSE 3.18 Administration Guide 18
Preface
Welcometothesyslog-ngOpenSourceEdition3.18AdministratorGuide!
Thisdocumentdescribeshowtoconfigureandmanagesyslog-ng.Backgroundinformationforthetechnologyandconceptsusedbytheproductisalsodiscussed.
Summary of contents
Introductiontosyslog-ngdescribesthemainfunctionalityandpurposeofsyslog-ngOSE.
Theconceptsofsyslog-ngdiscussesthetechnicalconceptsandphilosophiesbehindsyslog-ngOSE.
Installingsyslog-ngdescribeshowtoinstallsyslog-ngOSEonvariousUNIX-basedplatformsusingtheprecompiledbinaries.
Thesyslog-ngOSEquick-startguideprovidesabrieflyexplainshowtoperformthemostcommonlogcollectingtaskswithsyslog-ngOSE.
Thesyslog-ngOSEconfigurationfilediscussestheconfigurationfileformatandsyntaxindetail,andexplainshowtomanagelarge-scaleconfigurationsusingincludedfilesandreusableconfigurationsnippets.
source:Read,receive,andcollectlogmessagesexplainshowtocollectandreceivelogmessagesfromvarioussources.
destination:Forward,send,andstorelogmessagesdescribesthedifferentmethodstostoreandforwardlogmessages.
log:Filterandroutelogmessagesusinglogpaths,flags,andfiltersexplainshowtorouteandsortlogmessages,andhowtousefilterstoselectspecificmessages.
Globaloptionsofsyslog-ngOSEliststheglobaloptionsofsyslog-ngOSEandexplainshowtousethem.
TLS-encryptedmessagetransfershowshowtosecureandauthenticatelogtransportusingTLSencryption.
templateandrewrite:Format,modify,andmanipulatelogmessagesdescribeshowtocustomizemessageformatusingtemplatesandmacros,howtorewriteandmodifymessages,andhowtouseregularexpressions.
parser:Parseandsegmentstructuredmessagesdescribeshowtosegmentandprocessstructuredmessageslikecomma-separatedvalues.
db-parser:Processmessagecontentwithapatterndatabase(patterndb)explainshowtoidentifyandprocesslogmessagesusingapatterndatabase.
Correlatinglogmessagesexplainshowtocorrelatelogmessagesthatmatchasetoffiltersorthatareidentifiedusingapatterndatabase.
syslog-ng OSE 3.18 Administration Guide
Preface19
Enrichinglogmessageswithexternaldataexplainshowtoimportdatafromexternalsourcestoincludeinthelogmessages,thusextending,enriching,andcomplementingthedatafoundinthelogmessage.
Statisticsofsyslog-ngdetailstheavailablestatisticsthatsyslog-ngOSEcollectsabouttheprocessedlogmessages.
Multithreadingandscalinginsyslog-ngOSEdescribeshowtoconfiguresyslog-ngOSEtousemultipleprocessors,andhowtooptimizeitsperformance.
Troubleshootingsyslog-ngofferstipstosolvingproblems.
Bestpracticesandexamplesgivesrecommendationstoconfigurespecialfeaturesofsyslog-ngOSE.
Thesyslog-ngmanualpagescontainsthemanualpagesofthesyslog-ngOSEapplication.
Third-partycontributionsincludesthetextofthelicensesapplicabletosyslog-ngOpenSourceEdition.
CreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseincludesthetextoftheCreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseapplicabletoThesyslog-ngOpenSourceEdition3.18AdministratorGuide.
Target audience and prerequisites
Thisguideisintendedforsystemadministratorsandconsultantsresponsiblefordesigningandmaintainingloggingsolutionsandlogcenters.ItisalsousefulforITdecisionmakerslookingforatooltoimplementcentralizedlogginginheterogeneousenvironments.
Thefollowingskillsandknowledgearenecessaryforasuccessfulsyslog-ngadministrator:
l Atleastbasicsystemadministrationknowledge.
l Anunderstandingofnetworks,TCP/IPprotocols,andgeneralnetworkterminology.
l WorkingknowledgeoftheUNIXorLinuxoperatingsystem.
l In-depthknowledgeoftheloggingprocessofvariousplatformsandapplications.
l Anunderstandingofthelegacysyslog(BSD-syslog)protocolandthenewsyslog(IETF-syslog)protocolstandard.
Products covered in this guide
Thisguidedescribestheuseofthefollowingproducts:
l syslog-ngOpenSourceEdition(syslog-ngOSE)3.18.1andlater
syslog-ng OSE 3.18 Administration Guide
Preface20
https://www.ietf.org/rfc/rfc3164.txthttps://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424
Summary of changes
ThissectionliststhechangesofThesyslog-ngOpenSourceEditionAdministratorGuide.
Version 3.17 - 3.18
Changes in product:
l Startingwithsyslog-ngOSEversion3.18,youcanwritecustommessagesourcesinPython.Bothserver-styleandfetcher-stylesourcesaresupported.Formoredetails,see"python:writingserver-stylePythonsources"intheAdministrationGuideand"python-fetcher:writingfetcher-stylePythonsources"intheAdministrationGuide.
l Thehttp()destinationcannowsendabatchoflogmessagesinasingleHTTPrequest,greatlyimprovingtheperformance.Inaddition,thisfeaturealsoallowsyoutopostproperJSON-encodedarraysasPOSTpayloads,whichisrequiredbyseveralRESTAPIs.Fordetails,seeAdministrationGuide.
l Whenhdfs-append-enabledissettotrue,syslog-ngOSEwillappendnewdatatotheendofanalreadyexistingHDFSfile.Notethatinthiscase,archivingisautomaticallydisabled,andsyslog-ngOSEwillignorethehdfs-archive-diroption.
l Thehdfsdestinationnowsupportsthetime-reap()option.Fordetails,see"HDFSdestinationoptions"intheAdministrationGuide.
l Newtemplatefunctionsareavailable:url-decode()andbase64-encode().Fordetails,see"Templatefunctionsofsyslog-ngOSE"intheAdministrationGuide.
l Thesyslog-ng-ctl configcommandcandisplaythecontentsoftheconfigurationfilethatsyslog-ngOSEiscurrentlyrunning.
l Therekeyoptionofvalue-pairs()nowsupportsanewtransformation:shift-levels.Itcutsdot-delimited"levels"inthename(includingtheinitialdot).Forexample,--shift-levels 2deletestheprefixuptotheseconddotinthenameofthekey:.iptables.SRCbecomesSRC
Fordetails,see"value-pairs()"intheAdministrationGuide.
l Thevalue-pairs()optionnowhasanewscope:none.Thisscoperesetspreviouslyaddedscopes,makingitpossibletogetremoveautomaticallyaddedname-valuepairsfromthescope.
Fordetails,see"value-pairs()"intheAdministrationGuide.
l Themax-channelandframe-sizeoptionshavebeenaddedtotheamqp()destination.
Changes in documentation:
l Extendingsyslog-ngOSEinPythonhasbeensupportedforseveralreleases,butsofarthisfeaturewasmostlyundocumented.Nowyoucanfindmoredetailsaboutthis
syslog-ng OSE 3.18 Administration Guide
Preface21
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide//https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/hdfs-storing-messages-on-the-hadoop-distributed-file-system-hdfs/hdfs-destination-options/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/hdfs-storing-messages-on-the-hadoop-distributed-file-system-hdfs/hdfs-destination-options/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/template-and-rewrite-format-modify-and-manipulate-log-messages/customize-message-format-using-macros-and-templates/template-functions-of-syslog-ng-ose/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/amqp-publishing-messages-using-amqp/amqp-destination-options/
featurein"python:writingcustomPythondestinations"intheAdministrationGuide.
Version 3.16 - 3.17
Changes in product:
l Anewsourcedriver,linux-audit(),hasbeenadded.Thelinux-audit()sourcereadsandautomaticallyparsestheLinuxauditlogs.Fordetails,seelinux-audit:CollectingmessagesfromLinuxauditlogs.
l Anewsystemsourceoption,exclude-kmsg()makesitpossibletoavoidduplicatecollectionofkernellogsorerrorsinkernellogcollection(forexample,inscenarioswherethelogmanagementonthehostsystemandthecontainerizedsolutionarecollectingthekernellogssimultaneously).Whensettoyes,syslog-ngOSEwillomitkernellogsonplatformswheretheyareavailableseparately.
l Youcannowrefertoanyadditionalparametersattheendoftheargumentinablockbyaddingthreedotstoit().Ittellssyslog-ngOSEthatthismacroaccepts`__VARARGS__`,thereforeanyname-valuepaircanbepassedwithoutvalidation.Fordetails,seePassingargumentstoconfigurationblocks.
l Youcannowmakeparametersmandatoryinblockdefinitionsbydefiningthemwithemptybrackets().Fordetails,seeMandatoryparameters.
l Thefailover()optionallowsyoutospecifywhathappensaftersyslog-ngOSEfailsovertoasecondaryserver.Additionally,thefailover-servers()optionhasbeendeprecatedandremovedfromthedocument.Formoreinformationaboutthefailover()option,seeClient-sidefailoveronpage558.
l AddedsupportforthetimestampformatusedbyCiscoUnifiedCallManagerintheCiscoParser.Fordetails,seethesourcecodeofthisparseronGitHub.
Changes in documentation:
l AnoteaboutJVMstillrunningafterdeletingallJavadestinationsandreloadingsyslog-nghasbeenaddedtothedescriptionofJavadestinations.
l Thedefaultvalueofthe--skip-tokensparameteroftheloggenapplicationhasbeenchangedto0.Fordetails,seeTheloggenmanualpage.
Version 3.15 - 3.16
Changes in product:
l Anewdestinationdriver,telegram(),hasbeenadded.Thetelegram()destinationsendslogmessagestoTelegram,whichisasecure,cloud-basedmobileanddesktop
syslog-ng OSE 3.18 Administration Guide
Preface22
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/python-writing-custom-python-destinations/https://github.com/balabit/syslog-ng/blob/master/scl/cisco/plugin.confhttps://core.telegram.org/
messagingapp.Formoreinformation,seeTelegram:SendingmessagestoTelegram.
l Anewtemplatefunction,urlencode,hasbeenadded.Youcanusetheurlencodetemplatefunctiontogetherwiththetelegram()destinationtosendsyslogmessagestoTelegram.Formoreinformation,seeTemplatefunctionsofsyslog-ngOSE.
l Toensurethatamoduleisloaded,[email protected],seeLoadingmodules.
l Theadd-contextual-data()hasbeenextendedwiththeignore-case()option.Formoreinformation,seeOptionsadd-contextual-data().
l Thehook-commands()hasbeenadded,whichmakesitpossibletoexecuteexternalprogramswhentheyareinitializedortorndown.Thehook-commands()canbeusedforbothsourceanddestinationdrivers.Formoreinformation,seehook-commands().
Version 3.14 - 3.15
Changes in product:
l Itisnowpossibletouseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeif-else-elif:Conditionalexpressions.
l Anewlogpathflag,drop-unmatched,hasbeenadded.Thenewflagcausesmessagestobedroppedalongalogpathwhentheydonotmatchafilterorarediscardedbyaparser.Fordetails,seeLogpathflags.
l SupportforElasticsearch'sShieldhasbeenremoved.
l SupportforPOSIXregularexpressionshasbeenremoved.
Version 3.13 - 3.14
Changes in product:
l Youcanusepassword-protectedprivatekeysinthenetwork()andsyslog()sourceanddestinationdrivers.Fordetails,seePassword-protectedkeys.
l Tobettercontroltowhichlogmessagesyouaddcontextualdata,youcanusefiltersasselectors.Inthiscase,thefirstcolumnoftheCSVdatabasefilemustcontainthenameofafilter.Foreachmessage,syslog-ngOSEevaluatesthefiltersintheordertheyappearinthedatabasefile.Ifafiltermatchesthemessage,syslog-ngOSEaddsthename-valuepairrelatedtothefilter.Fordetails,seeUsingfiltersasselector.
syslog-ng OSE 3.18 Administration Guide
Preface23
https://core.telegram.org/
Version 3.12 - 3.13
Changes in product:
l Anewsourcedriver,stdin(),hasbeenadded.Thestdin()drivercollectsmessagesfromthestandardinputstream.Formoreinformation,seestdin:Collectingmessagesfromthestandardinputstream.
l Anewdestination,SendinglogstoGraylog,andatemplatetosendsyslogmessagestoGraylog,format-gelf,hasbeenadded.
l Anewtemplatefunction,getent,hasbeenadded.YoucanusethegetenttemplatefunctiontolookupentriesfromtheNameServiceSwitchlibraries.Formoreinformation,seegetent.
l Thedefaultvaluesofthe--enable-json,--enable-mongodb,and--with-libmongo-clientcompileparametershavechanged.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.
l Anewcompileoption,--with-module-path,hasbeenadded.Thenewoptionspecifiessyslog-ngOSE'smoduleinstallationdirectory.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.
l Anewdestinationdriver,osquery(),hasbeenadded.Thenewdriversendslogmessagestoosquery'ssyslogtable.Formoreinformation,seeosquery:Sendinglogmessagestoosquery'ssyslogtable.
l ItisnowpossibletospecifyTLSoptionsinatls()block.Formoreinformation,see:
l amqp()destinationoptions
l HTTPdestinationoptions
l riemann()destinationoptions
l SupportformicrosecondsinRiemanndestinationshasbeenintroduced.Formoreinformation,seeevent-time().
l Moduleauto-loadingnowalsoworksforthesystem()source.Formoreinformation,see--default-modules.
Changes in documentation:
l Anewsectiondescribingcommonerrormessageshasbeenaddedtothedocument.Formoreinformation,seeErrormessages.
l Severalcorrectionsandeditorialchanges.
syslog-ng OSE 3.18 Administration Guide
Preface24
Version 3.11 - 3.12
Changes in product:
l Anewsystemd-journal()sourceoption,calledread-old-records(),hasbeenadded.Formoreinformation,seeread-old-records().
l Anoptioncalledjvm-options()hasbeenadded,whichallowsyoutofine-tuneJavaVirtualMachinesettingswhenconfiguringElasticsearch,HDFS,andApacheKafkadestinations,orwebservicestowhichyousendlogmessagesviatheHTTPprotocol.Fordetails,see:
l Elasticsearchdestinationoptions
l Elasticsearch2destinationoptions
l HDFSdestinationoptions
l HTTPdestinationoptions
l Kafkadestinationoptions
l Globaloptions
l AnewHDFSdestinationoption,calledhdfs-append-enabled()hasbeenadded.Forfurtherinformation,seehdfs-append-enabled().
l Macrosarenowsupportedinthehdfs-file()option.Fordetails,seehdfs-file().
l ThefollowingnewTLSoptionshavebeenadded:
l dhparam-file()
l ecdh-curve-list()
l pkcs12-file().
l Anewparser,capableofprocessinginputinXMLformat,hasbeenadded.Formoreinformation,seeTheXMLparser.
Changes in documentation:
l Addedsectionaboutcommercialversionofsyslog-ng.Formoreinformation,seeCommercialversionofsyslog-ng.
l Addedwarningabouttherequirementtodeletethepersistfileoncethedir()optionofdisk-buffer()hasbeenmodifiedoranewonehasbeenadded.Formoreinformation,seedestination:Forward,send,andstorelogmessages.
l ClarifiedinformationaboutthePythonparser'sdeinit()method.Itrunsnotonlyatasyslog-nggracefulstop,butatareloadtoo.Fordetails,seeMethodsofthepython()parser.
l Severalcorrectionsandeditorialchanges.
syslog-ng OSE 3.18 Administration Guide
Preface25
Version 3.10 - 3.11
Changes in product:
l LookingupGeoIP2datafromIPaddresseshasbeenaddedtothedocument.
l http:PostingmessagesoverHTTPwithoutJavahasbeenupgradedwithnewimprovements.
l Thegeoip()parserisnowdeprecated.LookingupGeoIPdatafromIPaddresses(DEPRECATED).
l Thetemplate()optionhasbeenaddedtotheApacheAccessLogParser.Fordetails,see:TheApacheAccessLogParser.
l SSL-relatedoptionshavebeenaddedtoamqp()destination.Fordetails,see:amqp()destinationoptions.
l Theprefix()optionhasbeenaddedtotheCiscoparser.Fordetails,see:TheCiscoParser.
l Thedrop-unmatched()optionhasbeenaddedtothedb-parser()statement.Fordetails,see:Usingpatterndatabases.
l Theevent-time()optionhasbeenaddedtotheRiemanndestination.Fordetails,see:riemann:MonitoringyourdatawithRiemann.
Changes in documentation:
l Anewexamplehasbeenaddedtotheosquery()source.Fordetails,see:osquery:Collectandparseosqueryresultlogs.
l Severalcorrectionsandeditorialchanges.
Version 3.9 - 3.10
Changes in product:
l wildcard-file: Collectingmessages frommultiple text files has been added tothe document.
l snmptrap:ReadNet-SNMPtrapshasbeenaddedtothedocument.
l osquery:Collectandparseosqueryresultlogshasbeenaddedtothedocument.
l Theelasticsearch2()destinationnowsupportsHTTPSmode,includingencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeelasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher.
l Thehttp()destinationnowsupportsencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeHTTPdestinationoptions.
syslog-ng OSE 3.18 Administration Guide
Preface26
l Thehdfs()destinationnowsupportsKerberosauthentication.Fordetails,seeKerberosauthenticationwithsyslog-nghdfs()destination.
l ThePythonParserhasbeenaddedtothedocument.
l TheCiscoParserhasbeenaddedtothedocument.
l map-value-pairs: Rename value-pairs to normalize logs has been added to thedocument.
l Thelist-*templatefunctionsallowyoutomanipulatecomma-separatedlists.Fordetails,seeListmanipulation.
l Thenewbasename()anddirname()templatefunctionsallowyoutoeasilyseparatethepathandfilenames.Fordetails,seeTemplatefunctionsofsyslog-ngOSE.
l stardatehasbeenaddedtothedocument.
l create-statement-append()hasbeenaddedtothedocument.
l Thedefaultvalueofthelog-msg-size()optionhasbeenincreasedto64k.Thatwaysyslog-ngOSEwillnottruncatelonglogmessages,whicharegettingincreasinglycommon.
Changes in documentation:
l Splunk:SendinglogmessagestoSplunkhasbeenaddedtothedocument.
l Aboutdiskqueuefileshasbeenaddedtothedocument.
l AnexamplefailurescripthasbeenaddedtoRunningafailurescript.
l Severalcorrectionsandeditorialchanges.
Version 3.8 - 3.9
Changes in product:
l WhenusingTLS-transport,youcannowusecertainfieldsoftheX.509certificatesasmacros.Fordetails,see.TLS.X509.
l Theelastic2()destinationdrivernowsupportsSearchGuard,analternativesecuritysolutionforElasticsearch.Fordetails,seeSearchGuardandsyslog-ngOSE.
l .TLS.X509hasbeenaddedtothedocument.
l Unsettingmessagefieldshasbeenupdatedwithgroupunset().
Changes in documentation:
l Correctionsandeditorialchanges.
syslog-ng OSE 3.18 Administration Guide
Preface27
https://github.com/floragunncom/search-guard
Version 3.7 - 3.8
Changes in product:
l Enrichinglogmessageswithexternaldatahasbeenaddedtothedocument.
l Correlatinglogmessageshasbeenaddedtothedocument.
l elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigherhasbeenaddedtothedocument.
l http:PostingmessagesoverHTTPwithoutJavahasbeenaddedtothedocument.
l logmatic:UsingLogmatic.iohasbeenaddedtothedocument.
l loggly:UsingLogglyhasbeenaddedtothedocument.
l Disk-basedbufferinghasbeenaddedtosyslog-ngOSE.Fordetails,seeUsingdisk-basedandmemorybuffering.
l What'snewinthesyslog-ngpatterndatabaseformatV5,,hasbeenaddedtoElement:create-contexthasbeenaddedtodb-parser:Processmessagecontentwithapatterndatabase(patterndb).
l Parsingdatesandtimestampshasbeenaddedtoparser:Parseandsegmentstructuredmessages.
l TheApacheAccessLogParserhasbeenaddedtoparser:Parseandsegmentstructuredmessages.
l Newoptionsoftheset()rewriteoperatorhavebeenaddedtoSettingmessagefieldstospecificvalues.
l ArewriteoperatortounsetfieldshasbeenaddedtoUnsettingmessagefields.
l Atemplatefunctionthatformatsname-valuepairsasArcSightCommonEventFormatextensionhasbeenaddedtoformat-cef-extension.
l NumericaltemplatefunctionsthatworkonnumericalvaluesofacorrelationcontexthavebeenaddedtoNumericaloperations.
l Theinherit-environment()optionhasbeenaddedtoprogram:Receivingmessagesfromexternalapplicationsandprogram:Sendingmessagestoexternalapplications.
l @NLSTRING@hasbeenaddedtoUsingpatternparsers.
Changes in documentation:
l LookingupGeoIPdatafromIPaddresses(DEPRECATED)hasbeenmovedtoEnrichinglogmessageswithexternaldata.
l Severalcorrectionsandeditorialchanges.
syslog-ng OSE 3.18 Administration Guide
Preface28
Version 3.6 - 3.7
Changes in product:
l mbox: Converting local e-mailmessages to logmessages has been added tothe document.
l Thekeep-alive()optionhasbeenaddedtotheprogram()destination.
l The Linux Audit Parser has been added to parser: Parse and segmentstructuredmessages.
l pythonhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.
l PostingmessagesoverHTTPhasbeenaddedtothedocument.
l Write your own custom destination in Java or Python has been added to thedocument.
l Looking up GeoIP data from IP addresses (DEPRECATED) has been added tothe document.
l Elasticsearchdestinationoptionshasbeenaddedtothedocument.
l kafka:PublishingmessagestoApacheKafkahasbeenaddedtothedocument.
l hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS)hasbeenaddedtothedocument.
l Parsingkey=valuepairshasbeenaddedtothedocument.
l format-cimhasbeenaddedtothedocument.
l Simpletemplatescanbedefinedwithoutbraces.Templatescanalsoreferenceothertemplates.Fordetails,seeTemplatesandmacros.
l Customtemplatefunctionscanbedefinedinthesyslog-ngOSEconfiguration.Fordetails,seeUsingtemplatefunctions.
l CSV-parserscanusestringsasdelimiters.Fordetails,seedelimiters().
l IPv6addressescanbefilteredusinganewfilter.Fordetails,seenetmask6().
l Theloggenutilitycansendmessagesindefinitelyusingthe--permanentoption.
l Thessl-options()optionhasbeedaddedtoTLSoptions.
l TLS-supporthasbeenaddedtoriemann()destinationoptions.
l Theextract-solaris-msgid()parserhasbeedaddedtosun-streams:CollectingmessagesonSunSolaris.
l Thecontextoptionofinherit-propertieshasbeedaddedtoActionsandmessagecorrelation.
l flush-lines()hasbeenaddedtothedocument.
l Thesanitize-utf8flaghasbeenaddedtothelistofsourceflags.
l Theformat-welffunctionhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.
syslog-ng OSE 3.18 Administration Guide
Preface29
l The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.
l Theuse-uniqid()optionhasbeenaddedtoGlobaloptionsofsyslog-ngOSE.
l TheUNIQIDmacrohasbeenaddedtoMacrosofsyslog-ngOSE.
l TheJSON-parsernowhandlesspecialcharactersinobjectnames.Fordetails,seeextract-prefix().
l Thesyslog-debuntoolusedtogeneratesyslog-ngOSEdebugbundleshasbeendocumented.Fordetails,seeThesyslog-ng-debunmanualpage.
l The--controloptionhasbeenaddedtotheThesyslog-ngmanualpagemanualpage.
l Version3.7andnewerautomaticallyincludestheplugin.conffilesfromthe/scl/*/directories,makingiteasiertouseanddistributeconfigurationblocks.
l The--enable-all-modulescompileroptionhasbeedaddedtoCompilingoptionsofsyslog-ngOSE.
l Thecreate-dirs()optionhasbeenaddedtounix-stream()andunix-dgram()destinationoptions.
Changes in documentation:
l Generatingconfigurationblocksfromascripthasbeenaddedtothedocument.
l Example:Sendingalertwhenaclientdisappearshasbeenaddedtothedocument.
l Thetcp(),tcp6(),udp(),udp6()sourceanddestinationdrivershavebeendeprecated,asalloftheirfunctionalitycanbeachievedwiththenetwork()driver.Forhelponmigratingtothenetwork()driver,seeChangeanoldsourcedrivertothenetwork()driverandChangeanolddestinationdrivertothenetwork()driver.
l ThebeginningofTroubleshootingsyslog-nghasbeenextendedwithbasictroubleshootinginformation.
l Thedescriptionofthechain-hostnames()globaloptionhasbeenclarifiedandextended.Fordetails,seechain-hostnames().
l Othereditorialcorrections.
Version 3.5 - 3.6
Changes in product:
Changes in documentation:
l riemann:MonitoringyourdatawithRiemannhasbeenaddedtothedocument.
l nodejs:ReceivingJSONmessages fromnodejsapplicationshasbeenadded tothedocument.
syslog-ng OSE 3.18 Administration Guide
Preface30
l systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstoragehasbeenaddedtothedocument.
l systemd-syslog:Collectingsystemdmessagesusingasockethasbeenaddedtothedocument.
l use-rcptid()hasbeenaddedtothedocument.
l Settingmultiplemessagefieldstospecificvalueshasbeenaddedtothedocument.
l TheretriesandthrottleoptionsareavailablefortheSMTP,MongoDB,AMQP,andRedisdestinations.
l Thedescriptionofthemulti-line-modeoptionhasbeenupdated.
l UNIXcredentialsandothermetadatahasbeenaddedtothedocument.
l RUNIDhasbeenaddedtoMacrosofsyslog-ngOSE.
l Theextract-prefixoptionhasbeenaddedtoTheJSONparserTheJSONparser.
l Thegraphite-output,orandpaddingtemplatefunctionshavebeenaddedtoTemplatefunctionsofsyslog-ngOSE.
l PCREisnowarequireddependencyofsyslog-ngOSE,andbydefault,syslog-ngOSEusesPCRE-styleregularexpressions.Therefore,the--enable-pcrecompliationoptionhasbeenremoved.
l graphite:SendingmetricstoGraphitehasbeenaddedtothedocument.
l pseudofile()hasbeenaddedtothedocument.
l Thecustom-domain()andstats-lifetime()optionshavebeenaddedtoGlobaloptions.
l Theretry_sql_insertsoptionhasbeenrenamedtoretriestoincreaseconsistency.
l on-error()canbesetlocallyforMongoDBdestinationsaswell.Also,MongoDBdestinationssupporttheusernameandpasswordoptions,andconnectingtotheserverusingUNIXdomainsockets.Fordetails,seemongodb:StoringmessagesinaMongoDBdatabase.
l Howsyslog-ngOSEconnectstheMongoDBserverhasbeenaddedtothedocument.
l Severaltyposandsyntaxerrorsinexampleshavebeencorrected.
Acknowledgments
OneIdentitywouldliketoexpressitsgratitudetothesyslog-ngusersandthesyslog-ngcommunityfortheirinvaluablehelpandsupport.
syslog-ng OSE 3.18 Administration Guide
Preface31
3
Introduction to syslog-ng
Thischapterintroducesthesyslog-ngOpenSourceEditionapplicationinanon-technicalmanner,discussinghowandwhyisituseful,andthebenefitsitofferstoanexistingITinfrastructure.
What syslog-ng is
Thesyslog-ngapplicationisaflexibleandhighlyscalablesystemloggingapplicationthatisidealforcreatingcentralizedandtrustedloggingsolutions.Amongothers,syslog-ngOSEallowsyouthefollowing.
Secure and reliable log transfer
Thesyslog-ngOSEapplicationenablesyoutosendthelogmessagesofyourhoststoremoteserversusingthelatestprotocolstandards.Youcancollectandstoreyourlogdatacentrallyondedicatedlogservers.TransferlogmessagesusingtheTCPprotocolensuresthatnomessagesarelost.
Disk-based message buffering
Tominimizetheriskoflosingimportantlogmessages,thesyslog-ngOSEapplicationcanstoremessagesonthelocalharddiskifthecentrallogserverorthenetworkconnectionbecomesunavailable.Thesyslog-ngapplicationautomaticallysendsthestoredmessagestotheserverwhentheconnectionisreestablished,inthesameorderthemessageswerereceived.Thediskbufferispersistentnomessagesarelostevenifsyslog-ngisrestarted.
Secure logging using TLS
Logmessagesmaycontainsensitiveinformationthatshouldnotbeaccessedbythirdparties.Therefore,syslog-ngOSEsupportstheTransportLayerSecurity(TLS)protocolto
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng32
encryptthecommunication.TLSalsoallowsyoutoauthenticateyourclientsandthelogserverusingX.509certificates.
Flexible data extraction and processing
Mostlogmessagesareinherentlyunstructured,whichmakesthemdifficulttoprocess.Toovercomethisproblem,syslog-ngOSEcomeswithasetofbuilt-inparsers,whichyoucancombinetobuildverycomplexthings.
Filter and classify
Thesyslog-ngOSEapplicationcansorttheincominglogmessagesbasedontheircontentandvariousparameterslikethesourcehost,application,andpriority.Youcancreatedirectories,files,anddatabasetablesdynamicallyusingmacros.Complexfilteringusingregularexpressionsandbooleanoperatorsoffersalmostunlimitedflexibilitytoforwardonlytheimportantlogmessagestotheselecteddestinations.
Parse and rewrite
Thesyslog-ngOSEapplicationcansegmentlogmessagestonamedfieldsorcolumns,andalsomodifythevaluesofthesefields.YoucanprocessJSONmessages,key-valuepairs,andmore.
Togetthemostinformationoutofyourlogdata,syslog-ngOSEallowsyoutocorrelatelogmessagesandaggregatetheextractedinformationintoasinglemessage.Youcanalsouseexternalinformationtoenrichyourlogdata.
Big data clusters
Thelogdatathatyourorganizationhastoprocess,store,andreviewincreasesdaily,somanyorganizationsusebigdatasolutionsfortheirlogs.Toaccomodatethishugeamountofdata,syslog-ngOSEnativelysupportsstoringlogmessagesinHDFSfilesandElasticsearchclusters.
Message queue support
Largeorganizationsincreasinglyrelyonqueuinginfrastructuretotransfertheirdata.syslog-ngOSEsupportsApacheKafka,theAdvancedMessageQueuingProtocol(AMQP),andtheSimpleTextOrientedMessagingProtocol(STOMP).
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng33
SQL, NoSQL, and monitoring
Storing your log messages in a database allows you to easily search and query themessages and interoperate with log analyzing applications. The syslog-ng applicationsupports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, andSQLite.
syslog-ngOSEalsoallowsyoutoextracttheinformationyouneedfromyourlogdata,anddirectlysendittoyourGraphite,Redis,orRiemannmonitoringsystem.
Wide protocol and platform support
syslog protocol standards
syslog-ngnotonlysupportslegacyBSDsyslog(RFC3164)andtheenhancedRFC5424protocolsbutalsoJavaScriptObjectNotation(JSON)andjournaldmessageformats.
Heterogeneous environments
Thesyslog-ngOSEapplicationistheidealchoicetocollectlogsinmassivelyheterogeneousenvironmentsusingseveraldifferentoperatingsystemsandhardwareplatforms,includingLinux,Unix,BSD,SunSolaris,HP-UX,Tru64,andAIX.
IPv4 and IPv6 support
Thesyslog-ngapplicationcanoperateinbothIPv4andIPv6networkenvironments,andcanreceiveandsendmessagestobothtypesofnetworks.
What syslog-ng is not
Thesyslog-ngapplicationisnotloganalysissoftware.Itcanfilterlogmessagesandselectonlytheonesmatchingcertaincriteria.Itcanevenconvertthemessagesandrestructurethemtoapredefinedformat,orparsethemessagesandsegmentthemintodifferentfields.Butsyslog-ngcannotinterpretandanalyzethemeaningbehindthemessages,orrecognizepatternsintheoccurrenceofdifferentmessages.
Why is syslog-ng needed?
Logmessagescontaininformationabouttheeventshappeningonthehosts.Monitoringsystemeventsisessentialforsecurityandsystemhealthmonitoringreasons.
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng34
Theoriginalsyslogprotocolseparatesmessagesbasedonthepriorityofthemessageandthefacilitysendingthemessage.Thesetwoparametersaloneareofteninadequatetoconsistentlyclassifymessages,asmanyapplicationsmightusethesamefacility,andthefacilityitselfisnotevenincludedinthelogmessage.Tomakethingsworse,manylogmessagescontainunimportantinformation.Thesyslog-ngapplicationhelpsyoutoselectonlythereallyinterestingmessages,andforwardthemtoacentralserver.
Companypoliciesorotherregulationsoftenrequirelogmessagestobearchived.Storingtheimportantmessagesinacentrallocationgreatlysimplifiesthisprocess.
What is new in syslog-ng Open Source Edition 3.18?
Version3.18ofsyslog-ngOpenSourceEditionincludesthefollowingmainfeatures.
Batch support in the http() destination driver
Thehttp()destinationcannowsendabatchoflogmessagesinasingleHTTPrequest,greatlyimprovingtheperformance.Inaddition,thisfeaturealsoallowsyoutopostproperJSON-encodedarraysasPOSTpayloads,whichisrequiredbyseveralRESTAPIs.Fordetails,seeAdministrationGuide.
Write your own destination in Python
Extendingsyslog-ngOSEinPythonhasbeensupportedforseveralreleases,butsofarthisfeaturewasmostlyundocumented.Nowyoucanfindmoredetailsaboutthisfeaturein"python:writingcustomPythondestinations"intheAdministrationGuide.
Write your own message source in Python
Startingwithsyslog-ngOSEversion3.18,youcanwritecustommessagesourcesinPython.Bothserver-styleandfetcher-stylesourcesaresupported.Formoredetails,see"python:writingserver-stylePythonsources"intheAdministrationGuideand"python-fetcher:writingfetcher-stylePythonsources"intheAdministrationGuide.
Enhancements
l Whenhdfs-append-enabledissettotrue,syslog-ngOSEwillappendnewdatatotheendofanalreadyexistingHDFSfile.Notethatinthiscase,archivingisautomaticallydisabled,andsyslog-ngOSEwillignorethehdfs-archive-diroption.
l Thehdfsdestinationnowsupportsthetime-reap()option.
l Theurlencode()templatefunctionhasbeenrenamedtourl-encode().Also,thetelegram()destinationnowautomaticallyencodesthemessages.
l Newtemplatefunctionsareavailable:url-decode()andbase64-encode().Fordetails,
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng35
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide//https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/python-writing-custom-python-destinations/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/
see"Templatefunctionsofsyslog-ngOSE"intheAdministrationGuide.
l Thesyslog-ng-ctl configcommandcandisplaythecontentsoftheconfigurationfilethatsyslog-ngOSEiscurrentlyrunning.
l Therekeyoptionofvalue-pairs()nowsupportsanewtransformation:shift-levels.Itcutsdot-delimited"levels"inthename(includingtheinitialdot).Forexample,--shift-levels 2deletestheprefixuptotheseconddotinthenameofthekey:.iptables.SRCbecomesSRC
Fordetails,see"value-pairs()"intheAdministrationGuide.
l Thevalue-pairs()optionnowhasanewscope:none.Thisscoperesetspreviouslyaddedscopes,makingitpossibletogetremoveautomaticallyaddedname-valuepairsfromthescope.
Fordetails,see"value-pairs()"intheAdministrationGuide.
l Whenreceivingmessageswiththedefault-network-drivers()source,syslog-ngOSEnowautomaticallysetsthe${.app.name}name-valuepairtothenameoftheapplicationthatsentthelogmessage.
Fordetails,see"default-network-drivers:Receiveandparsecommonsyslogmessages"intheAdministrationGuide.
Deprecated features
Theelasticsearch()destinationhasbeendeprecated,becauseitsupportsonlyElasticSearchversion1.x,whichhasbeenEnd-of-LifesinceJanuary,2017.Usetheelasticsearch2()destinationinstead.
Who uses syslog-ng?
Thesyslog-ngapplicationisusedworldwidebycompaniesandinstitutionswhocollectandmanagethelogsofseveralhosts,andwanttostoretheminacentralized,organizedway.Usingsyslog-ngisparticularlyadvantageousfor:
l InternetServiceProviders
l Financialinstitutionsandcompaniesrequiringpolicycompliance
l Server,web,andapplicationhostingcompanies
l Datacenters
l Wideareanetwork(WAN)operators
l Serverfarmadministrators.
Supported platformsThesyslog-ngOpenSourceEditionapplicationishighlyportableandisknowntorunonawiderangeofhardwarearchitectures(x86,x86_64,SUNSparc,PowerPC32and64,
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng36
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/template-and-rewrite-format-modify-and-manipulate-log-messages/customize-message-format-using-macros-and-templates/template-functions-of-syslog-ng-ose/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/default-network-drivers-receive-and-parse-common-syslog-messages/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/default-network-drivers-receive-and-parse-common-syslog-messages/
Alpha)andoperatingsystems,includingLinux,BSD,Solaris,IBMAIX,HP-UX,MacOSX,Cygwin,Tru64,andothers.
l Thesourcecodeofsyslog-ngOpenSourceEditionisreleasedundertheGPLv2licenseandisavailableonGitHub.
l SeetheDownloadspageforbinarypackages.
syslog-ng OSE 3.18 Administration Guide
Introduction to syslog-ng37
https://github.com/balabit/syslog-nghttps://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
4
The concepts of syslog-ng
Thischapterdiscussesthetechnicalconceptsofsyslog-ng.
The philosophy of syslog-ng
Typically,syslog-ngisusedtomanagelogmessagesandimplementcentralizedlogging,wheretheaimistocollectthelogmessagesofseveraldevicesonasingle,centrallogserver.Thedifferentdevicescalledsyslog-ngclientsallrunsyslog-ng,andcollectthelogmessagesfromthevariousapplications,files,andothersources.Theclientssendallimportantlogmessagestotheremotesyslog-ngserver,whichsortsandstoresthem.
Logging with syslog-ngThesyslog-ngapplicationreadsincomingmessagesandforwardsthemtotheselecteddestinations.Thesyslog-ngapplicationcanreceivemessagesfromfiles,remotehosts,andothersources.
Logmessagesentersyslog-nginoneofthedefinedsources,andaresenttooneormoredestinations.
Sourcesanddestinationsareindependentobjects,log pathsdefinewhatsyslog-ngdoeswithamessage,connectingthesourcestothedestinations.Alogpathconsistsofoneormoresourcesandoneormoredestinations:messagesarrivingfromasourcearesenttoeverydestinationlistedinthelogpath.Alogpathdefinedinsyslog-ngiscalledalog statement.
Optionally,logpathscanincludefilters.Filtersarerulesthatselectonlycertainmessages,forexample,selectingonlymessagessentbyaspecificapplication.Ifalogpathincludesfilters,syslog-ngsendsonlythemessagessatisfyingthefilterrulestothedestinationssetinthelogpath.
Otheroptionalelements thatcanappear in logstatementsareparsersand rewriting rules.Parserssegmentmessages intodifferent fields tohelpprocessing themessages,while rewrite rulesmodify themessagesbyadding, replacing,or removingpartsofthemessages.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng38
The route of a log message in syslog-ng
Purpose:
Thefollowingprocedureillustratestherouteofalogmessagefromitssourceonthesyslog-ngclienttoitsfinaldestinationonthecentralsyslog-ngserver.
Figure 1: The route of a log message
Steps:
1. Adeviceorapplicationsendsalogmessagetoasourceonthesyslog-ngclient.Forexample,anApachewebserverrunningonLinuxentersamessageintothe/var/log/apachefile.
2. Thesyslog-ngclientrunningonthewebserverreadsthemessagefromits/var/log/apachesource.
3. Thesyslog-ngclientprocessesthefirstlogstatementthatincludesthe/var/log/apachesource.
4. Thesyslog-ngclientperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement,forexample,totheremotesyslog-ngserver.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng39
CAUTION:
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
NOTE:
Thesyslog-ngclientsendsamessagetoallmatchingdestinationsbydefault.Asaresult,amessagemaybesenttoadestinationmorethanonce,ifthedestinationisusedinmultiplelogstatements.Topreventsuchsituations,usethefinalflaginthedestinationstatements.Fordetails,seeLogstatementflags.
5. Thesyslog-ngclientprocessesthenextlogstatementthatincludesthe/var/log/apachesource,repeatingSteps3-4.
6. Themessagesentby thesyslog-ngclientarrives fromasourceset in thesyslog-ngserver.
7. Thesyslog-ngserverreadsthemessagefromitssourceandprocessesthefirstlogstatementthatincludesthatsource.
8. Thesyslog-ngserverperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement.
CAUTION:
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
9. Thesyslog-ngserverprocessesthenextlogstatement,repeatingSteps7-9.
NOTE:
Thesyslog-ngapplicationcanstopreadingmessagesfromitssourcesifthedestinationscannotprocessthesentmessages.Thisfeatureiscalledflow-controlandisdetailedinManagingincomingandoutgoingmessageswithflow-control.
Modes of operationThesyslog-ngOpenSourceEditionapplicationhasthreetypicaloperationscenarios:Client,Server,andRelay.
Client mode
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng40
Figure 2: Client-mode operation
Inclientmode,syslog-ngcollectsthelocallogsgeneratedbythehostandforwardsthemthroughanetworkconnectiontothecentralsyslog-ngserverortoarelay.Clientsoftenalsologthemessageslocallyintofiles.
Relay modeFigure 3: Relay-mode operation
Inrelaymode,syslog-ngreceiveslogsthroughthenetworkfromsyslog-ngclientsandforwardsthemtothecentralsyslog-ngserverusinganetworkconnection.Relaysalsologthemessagesfromtherelayhostintoalocalfile,orforwardthesemessagestothecentralsyslog-ngserver.
Server mode
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng41
Figure 4: Server-mode operation
Inservermode,syslog-ngactsasacentrallog-collectingserver.Itreceivesmessagesfromsyslog-ngclientsandrelaysoverthenetwork,andstoresthemlocallyinfiles,orpassesthemtootherapplications,forexampleloganalyzers.
Global objectsThesyslog-ngapplicationusesthefollowingobjects:
l Source driver:Acommunicationmethodusedtoreceivelogmessages.Forexample,syslog-ngcanreceivemessagesfromaremotehostviaTCP/IP,orreadthemessagesofalocalapplicationfromafile.Fordetailsonsourcedrivers,seesource:Read,receive,andcollectlogmessages.
l Source:Anamedcollectionofconfiguredsourcedrivers.
l Destination driver:Acommunicationmethodusedtosendlogmessages.Forexample,syslog-ngcansendmessagestoaremotehostviaTCP/IP,orwritethemessagesintoafileordatabase.Fordetailsondestinationdrivers,seedestination:Forward,send,andstorelogmessages.
l Destination:Anamedcollectionofconfigureddestinationdrivers.
l Filter:Anexpressiontoselectmessages.Forexample,asimplefiltercanselectthe
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng42
messagesreceivedfromaspecifichost.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Macro:Anidentifierthatreferstoapartofthelogmessage.Forexample,the${HOST}macroreturnsthenameofthehostthatsentthemessage.Macrosareoftenusedintemplatesandfilenames.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Parser:Parsersareobjectsthatparsetheincomingmessages,orpartsofamessage.Forexample,thecsv-parser()cansegmentmessagesintoseparatecolumnsatapredefinedseparatorcharacter(forexampleacomma).Everycolumnhasauniquenamethatcanbeusedasamacro.Fordetails,seeparser:Parseandsegmentstructuredmessagesanddb-parser:Processmessagecontentwithapatterndatabase(patterndb).
l Rewrite rule:Arulemodifiesapartofthemessage,forexample,replacesastring,orsetsafieldtoaspecifiedvalue.Fordetails,seeModifyingmessagesusingrewriterules.
l Log paths:Acombinationofsources,destinations,andotherobjectslikefilters,parsers,andrewriterules.Thesyslog-ngapplicationsendsmessagesarrivingfromthesourcesofthelogpathstothedefineddestinations,andperformsfiltering,parsing,andrewritingofthemessages.Logpathsarealsocalledlogstatements.Logstatementscanincludeother(embedded)logstatementsandjunctionstocreatecomplexlogpaths.Fordetails,seelog:Filterandroutelogmessagesusinglogpaths,flags,andfilters.
l Template:Atemplateisasetofmacrosthatcanbeusedtorestructurelogmessagesorautomaticallygeneratefilenames.Forexample,atemplatecanaddthehostnameandthedatetothebeginningofeverylogmessage.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Option:Optionssetglobalparametersofsyslog-ng,liketheparametersofnameresolutionandtimezonehandling.Fordetails,seeGlobaloptionsofsyslog-ngOSE.
Fordetailsontheaboveobjects,seeTheconfigurationsyntaxindetail.
Timezones and daylight savingThesyslog-ngapplicationreceivesthetimezoneanddaylightsavinginformationfromtheoperatingsystemitisinstalledon.Iftheoperatingsystemhandlesdaylightsavingcorrectly,sodoessyslog-ng.
Thesyslog-ngapplicationsupportsmessagesoriginatingfromdifferenttimezones.Theoriginalsyslogprotocol(RFC3164)doesnotincludetimezoneinformation,butsyslog-ngprovidesasolutionbyextendingthesyslogprotocoltoincludethetimezoneinthelogmessages.Thesyslog-ngapplicationalsoenablesadministratorstosupplytimezoneinformationforlegacydeviceswhichdonotsupporttheprotocolextension.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng43
How syslog-ng OSE assigns timezone to the message
Whensyslog-ngOSEreceivesamessage,itassignstimezoneinformationtothemessageusingthefollowingalgorithm.
1. Thesenderapplication(forexamplethesyslog-ngclient)orhostspecifiesthetimezoneofthemessages.Iftheincomingmessageincludesatimezoneitisassociatedwiththemessage.Otherwise,thelocaltimezoneisassumed.
2. Specifythetime-zone()parameterforthesourcedriverthatreadsthemessage.Thistimezonewillbeassociatedwiththemessagesonlyifnotimezoneisspecifiedwithinthemessageitself.Eachsourcedefaultstothevalueoftherecv-time-zone()globaloption.Itisnotpossibletooverrideonlythetimezoneinformationoftheincomingmessage,butsettingthekeep-timestamp()optiontonoallowssyslog-ngOSEtoreplacethefulltimestamp(timezoneincluded)withthetimethemessagewasreceived.
NOTE:
Whenprocessingamessagethatdoesnotcontaintimezoneinformation,thesyslog-ngOSEapplicationwillusethetimezoneanddaylight-savingthatwaseffectivewhenthetimestampwasgenerated.Forexample,thecurrenttimeis2011-03-11(March11,2011)intheEU/Budapesttimezone.Whendaylight-savingisactive(summertime),theoffsetis+02:00.Whendaylight-savingisinactive(wintertime)thetimezoneoffsetis+01:00.Ifthetimestampofanincomingmessageis2011-01-01,thetimezoneassociatedwiththemessagewillbe+01:00,butthetimestampwillbeconverted,because2011-01-01meantwintertimewhendaylightsavingisnotactivebutthecurrenttimezoneis+02:00.
3. Specifythetimezoneinthedestinationdriverusingthetime-zone()parameter.Eachdestinationdrivermighthaveanassociatedtimezonevalue:syslog-ngconvertsmessagetimestampstothistimezonebeforesendingthemessagetoitsdestination(fileornetworksocket).Eachdestinationdefaultstothevalueofthesend-time-zone()globaloption.
NOTE:
Amessagecanbesenttomultipledestinationzones.Thesyslog-ngapplicationconvertsthetimezoneinformationproperlyforeveryindividualdestinationzone.
CAUTION:
If syslog-ng OSE sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so syslog-ng OSE will convert the hour:min values based on the explicitly specified timezone.
4. Ifthetimezoneisnotspecified,localtimezoneisused.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng44
5. Whenmacroexpansionsareusedinthedestinationfilenames,thelocaltimezoneisused.(Also,ifthetimestampofthereceivedmessagedoesnotcontaintheyearofthemessage,syslog-ngOSEusesthelocalyear.)
A note on timezones and timestampsIf the clients run syslog-ng, then use the ISO timestamp, because it includestimezone information. Thatway you do not need to adjust the recv-time-zone()parameter of syslog-ng.
Ifyouwantsyslog-ngtooutputtimestampsinUnix(POSIX)timeformat,usetheS_UNIXTIMEandR_UNIXTIMEmacros.Youdonotneedtochangeanyofthetimezonerelatedparameters,becausethetimestampinformationofincomingmessagesisconvertedtoUnixtimeinternally,andUnixtimeisatimezone-independenttimerepresentation.(Actually,UnixtimemeasuresthenumberofsecondselapsedsincemidnightofCoordinatedUniversalTime(UTC)January1,1970,butdoesnotcountleapseconds.)
Product licensing
Startingwithversion3.2,thesyslog-ngOpenSourceEditionapplicationislicensedunderacombinedLGPL+GPLlicense.Thecoreofsyslog-ngOSEislicensedundertheGNULesserGeneralPublicLicenseVersion2.1license,whiletherestofthecodebaseislicensedundertheGNUGeneralPublicLicenseVersion2license.
NOTE:
Practically,thecodestoredunderthelibdirectoryofthesourcecodepackageisunderLGPL,therestisGPL.
FordetailsabouttheLGPLandGPLlicenses,seeGNULesserGeneralPublicLicenseandGNUGeneralPublicLicense,respectively.
High availability supportMultiplesyslog-ngserverscanberuninfail-overmode.Thesyslog-ngapplicationdoesnotincludeanyinternalsupportforthis,asclusteringsupportmustbeimplementedontheoperatingsystemlevel.AtoolthatcanbeusedtocreateUNIXclustersisHeartbeat(fordetails,seethispage).
The structure of a log messageThefollowingsectionsdescribethestructureoflogmessages.Currentlytherearetwostandardsyslogmessageformats:
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng45
http://www.linux-ha.org/wiki/Main_Page/
l TheoldstandarddescribedinRFC3164(alsocalledtheBSD-syslogorthelegacy-syslogprotocol):seeBSD-syslogorlegacy-syslogmessages
l ThenewstandarddescribedinRFC5424(alsocalledtheIETF-syslogprotocol):seeIETF-syslogmessages
l TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesbetweensyslog-ngnodes:seeEnterprise-widemessagemodel(EWMM)
l Howmessagesarerepresentedinsyslog-ngOSE:seeMessagerepresentationinsyslog-ngOSE.
BSD-syslog or legacy-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtothelegacy-syslogorBSD-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:
l PRI
l HEADER
l MSG
Thetotalmessagecannotbelongerthan1024bytes.
Thefollowingisasamplesyslogmessage:
Feb 25 14:09:07 webserver syslogd: restart
Themessagecorrespondstothefollowingformat:
timestamp hostname application: message
Thedifferentpartsofthemessageareexplainedinthefollowingsections.
NOTE:
Thesyslog-ngapplicationsupportslongermessagesaswell.Fordetails,seethelog-msg-size()optioninGlobaloptions.However,itisnotrecommendedtoenablemessageslargerthanthepacketsizewhenusingUDPdestinations.
The PRI message part
ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.
NOTE:
Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng46
https://tools.ietf.org/search/rfc3164https://tools.ietf.org/search/rfc3164
Numerical Code Facility
0 kernelmessages
1 user-levelmessages
2 mailsystem
3 systemdaemons
4 security/authorizationmessages
5 messagesgeneratedinternallybysyslogd
6 lineprintersubsystem
7 networknewssubsystem
8 UUCPsubsystem
9 clockdaemon
10 security/authorizationmessages
11 FTPdaemon
12 NTPsubsystem
13 logaudit
14 logalert
15 clockdaemon
16-23 locallyusedfacilities(local0-local7)
Table 1: syslog Message Facilities
Thefollowingtableliststheseverityvalues.
Numerical Code Severity
0 Emergency:systemisunusable
1 Alert:actionmustbetakenimmediately
2 Critical:criticalconditions
3 Error:errorconditions
4 Warning:warningconditions
5 Notice:normalbutsignificantcondition
6 Informational:informationalmessages
7 Debug:debug-levelmessages
Table 2: syslog Message Severities
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng47
The HEADER message partTheHEADERpartcontainsatimestampandthehostname(withoutthedomainname)ortheIPaddressofthedevice.ThetimestampfieldisthelocaltimeintheMmm dd hh:mm:ssformat,where:
l MmmistheEnglishabbreviationofthemonth:Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec.
l ddisthedayofthemonthontwodigits.Ifthedayofthemonthislessthan10,thefirstdigitisreplacedwithaspace.(ForexampleAug 7.)
l hh:mm:ssisthelocaltime.Thehour(hh)isrepresentedina24-hourformat.Validentriesarebetween00and23,inclusive.Theminute(mm)andsecond(ss)entriesarebetween00and59inclusive.
NOTE:
Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.Fordetails,seethets-format()optioninGlobaloptions.
The MSG message part
TheMSGpartcontainsthenameoftheprogramorprocessthatgeneratedthemessage,andthetextofthemessageitself.TheMSGpartisusuallyinthefollowingformat:program[pid]: message text.
IETF-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtotheIETF-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:
l HEADER(includesthePRIaswell)
l STRUCTURED-DATA
l MSG
Thefollowingisasamplesyslogmessage(source:https://tools.ietf.org/html/rfc5424):
1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
Themessagecorrespondstothefollowingformat:
VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
Inthisexample,theFacilityhasthevalueof4,severityis2,soPRIis34.TheVERSIONis1.Themessagewascreatedon11October2003at10:14:15pmUTC,3millisecondsintothenextsecond.Themessageoriginatedfromahostthatidentifiesitselfas"mymachine.example.com".TheAPP-NAMEis"su"andthePROCIDisunknown.The
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng48
https://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424
MSGIDis"ID47".TheMSGis"'suroot'failedforlonvick...",encodedinUTF-8.TheencodingisdefinedbytheBOM:
Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.
ThereisnoSTRUCTURED-DATApresentinthemessage,thisisindicatedby"-"intheSTRUCTURED-DATAfield.TheMSGis"'suroot'failedforlonvick...".
TheHEADERpartofthemessagemustbeinplainASCIIformat,theparametervaluesoftheSTRUCTURED-DATApartmustbeinUTF-8,whiletheMSGpartshouldbeinUTF-8.Thedifferentpartsofthemessageareexplainedinthefollowingsections.
The PRI message part
ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.
NOTE:
Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.
Numerical Code Facility
0 kernelmessages
1 user-levelmessages
2 mailsystem
3 systemdaemons
4 security/authorizationmessages
5 messagesgeneratedinternallybysyslogd
6 lineprintersubsystem
7 networknewssubsystem
8 UUCPsubsystem
9 clockdaemon
10 security/authorizationmessages
11 FTPdaemon
Table 3: syslog Message Facilities
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng49
Numerical Code Facility
12 NTPsubsystem
13 logaudit
14 logalert
15 clockdaemon
16-23 locallyusedfacilities(local0-local7)
Thefollowingtableliststheseverityvalues.
Numerical Code Severity
0 Emergency:systemisunusable
1 Alert:actionmustbetakenimmediately
2 Critical:criticalconditions
3 Error:errorconditions
4 Warning:warningconditions
5 Notice:normalbutsignificantcondition
6 Informational:informationalmessages
7 Debug:debug-levelmessages
Table 4: syslog Message Severities
The HEADER message partTheHEADERpartcontainsthefollowingelements:
l VERSION: Version number of the syslog protocol standard. Currently this canonly be 1.
l ISOTIMESTAMP:ThetimewhenthemessagewasgeneratedintheISO8601compatiblestandardtimestampformat(yyyy-mm-ddThh:mm:ss+-ZONE),forexample:2006-06-13T15:58:00.123+01:00.
l HOSTNAME:Themachinethatoriginallysentthemessage.
l APPLICATION:Thedeviceorapplicationthatgeneratedthemessage
l PID:TheprocessnameorprocessIDofthesyslogapplicationthatsentthemessage.It isnotnecessarilytheprocessIDoftheapplicationthatgeneratedthemessage.
l MESSAGEID:TheIDnumberofthemessage.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng50
NOTE:
Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.ThetimestampusedintheIETF-syslogprotocolisderivedfromRFC3339,whichisbasedonISO8601.Fordetails,seethets-format()optioninGlobaloptions.
Thesyslog-ngOSEapplicationwilltruncatethefollowingfields:
l IfAPP-NAMEislongerthan48charactersitwillbetruncatedto48characters.
l IfPROC-IDislongerthan128charactersitwillbetruncatedto128characters.
l IfMSGIDislongerthan32charactersitwillbetruncatedto32characters.
l IfHOSTNAMEislongerthan255charactersitwillbetruncatedto255characters.
The STRUCTURED-DATA message part
TheSTRUCTURED-DATAmessagepartmaycontainmeta-informationaboutthesyslogmessage,orapplication-specificinformationsuchastrafficcountersorIPaddresses.STRUCTURED-DATAconsistsofdatablocksenclosedinbrackets([]).EveryblockincludestheIDoftheblock,andoneormorename=valuepairs.Thesyslog-ngapplicationautomaticallyparsestheSTRUCTURED-DATApartofsyslogmessages,whichcanbereferencedinmacros(fordetails,seeMacrosofsyslog-ngOSE).AnexampleSTRUCTURED-DATAblocklookslike:
[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
The MSG message part
TheMSGpartcontainsthetextofthemessageitself.TheencodingofthetextmustbeUTF-8iftheBOM1characterispresentinthemessage.IfthemessagedoesnotcontaintheBOMcharacter,theencodingistreatedasunknown.UsuallymessagesarrivingfromlegacysourcesdonotincludetheBOMcharacter.CRLFcharacterswillnotberemovedfromthemessage.
Enterprise-wide message model (EWMM)ThefollowingsectiondescribesthestructureoflogmessagesusingtheEnterprise-widemessagemodelorEWMMmessageformat.
TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,through
1Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng51
anynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.
ThefollowingisasamplelogmessageinEWMMformat.
1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
Themessagehasthefollowingparts.
l TheheaderofthecomplieswiththeRFC5424messageformat,wherethePROGRAMfieldissetto@syslog-ng,andtheSDATAfieldisempty.
l TheMESSAGEpartisinJSONformat,andcontainstheactualmessage,aswellasanyname-valuepairsthatsyslog-ngOSEhasattachedtoorextractedfromthemessage.The${._TAGS}fieldcontainstheidentifierofthesyslog-ngsourcethathasoriginallyreceivedthemessageonthefirstsyslog-ngnode.
TosendamessageinEWMMformat,youcanusethesyslog-ng()destinationdriver,ortheformat-ewmm()templatefunction.
ToreceiveamessageinEWMMformat,youcanusethedefault-destination-drivers()sourcedriver,ortheewmm-parser()parser.
Message representation in syslog-ng OSEWhenthesyslog-ngOSEapplicationreceivesamessage,itautomaticallyparsesthemessage.Thesyslog-ngOSEapplicationcanautomaticallyparselogmessagesthatconformtotheRFC3164(BSDorlegacy-syslog)ortheRFC5424(IETF-syslog)messageformats.Ifsyslog-ngOSEcannotparseamessage,itresultsinanerror.
TIP:
Incaseyouneedtorelaymessagesthatcannotbeparsedwithoutanymodificationsorchanges,usetheflags(no-parse)optioninthesourcedefinition,andatemplatecontainingonlythe${MESSAGE}macrointhedestinationdefinition.
Toparsenon-syslogmessages,forexample,JSON,CSV,orothermessages,youcanusethebuilt-inparsersofsyslog-ngOSE.Fordetails,seeparser:Parseandsegmentstructuredmessages.
Aparsedsyslogmessagehasthefollowingparts.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng52
l Timestamps
Twotimestampsareassociatedwitheverymessage:oneisthetimestampcontainedwithinthemessage(thatis,whenthesendersentthemessage),theotheristhetimewhensyslog-ngOSEhasactuallyreceivedthemessage.
l Severity
Theseverityofthemessage.
l Facility
Thefacilitythatsentthemessage.
l Tags
Customtextlabelsaddedtothemessagethataremainlyusedforfiltering.Noneofthecurrentmessagetransportprotocolsaddstagstothelogmessages.Tagscanbeaddedtothelogmessageonlywithinsyslog-ngOSE.Thesyslog-ngOSEapplicationautomaticallyaddstheidofthesourceasatagtotheincomingmessages.Othertagscanbeaddedtothemessagebythepatterndatabase,orusingthetags()optionofthesource.
l IP address of the sender
TheIPaddressofthehostthatsentthemessage.NotethattheIPaddressofthesenderisahardmacroandcannotbemodifiedwithinsyslog-ngOSEbuttheassociatedhostnamecanbemodified,forexample,usingrewriterules.
l Hard macrosHardmacroscontaindatathatisdirectlyderivedfromthelogmessage,forexample,the${MONTH}macroderivesitsvaluefromthetimestamp.Themostimportantconsiderationwithhardmacrosisthattheyareread-only,meaningtheycannotbemodifiedusingrewriterulesorothermeans.
l Soft macrosSoftmacros(sometimesalsocalledname-valuepairs)areeitherbuilt-inmacrosautomaticallygeneratedfromthelogmessage(forexample,${HOST}),orcustomuser-createdmacrosgeneratedbyusingthesyslog-ngpatterndatabaseoraCSV-parser.TheSDATAfieldsofRFC5424-formattedlogmessagesbecomesoftmacrosaswell.Incontrastwithhardmacros,softmacrosarewritableandcanbemodifiedwithinsyslog-ngOSE,forexample,usingrewriterules.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng53
NOTE:
Itisalsopossibletosetthevalueofbuilt-insoftmacrosusingparsers,forexample,tosetthe${HOST}macrofromthemessageusingacolumnofaCSV-parser.
Thedataextractedfromthelogmessagesusingnamedpatternparsersinthepatterndatabasearealsosoftmacros.
TIP:
Forthelistofhardandsoftmacros,seeHardvs.softmacros.
Message size and encodingInternally,syslog-ngOSErepresentseverymessageasUTF-8.Themaximallengthofthelogmessagesislimitedbythelog-msg-size()option:ifamessageislongerthanthisvalue,syslog-ngOSEtruncatesthemessageatthelocationitreachesthelog-msg-size()value,anddiscardstherestofthemessage.
Whenencodingissetinasource(usingtheencoding()option)andthemessageislonger(inbytes)thanlog-msg-size()inUTF-8representation,syslog-ngOSEsplitsthemessageatanundefinedlocation(becausetheconversionbetweendifferentencodingsisnottrivial).
Structuring macros, metadata, and other value-pairs
Available in syslog-ng OSE 3.3 and later.
Thesyslog-ngOSEapplicationallowsyoutoselectandconstructname-valuepairsfromanyinformationalreadyavailableaboutthelogmessage,orextractedfromthemessageitself.Youcandirectlyusethisstructuredinformation,forexample,inthefollowingplaces:
l amqp()destination
l format-welf()templatefunction
l mongodb()destination
l stomp()destination
l orinotherdestinationsusingtheformat-json()templatefunction.
Whenusingvalue-pairs,therearethreewaystospecifywhichinformation(thatis,macrosorothername-valuepairs)toincludeintheselection.
l Selectgroupsofmacrosusingthescope()parameter,andoptionallyremovecertainmacrosfromthegroupusingtheexclude()parameter.
syslog-ng OSE 3.18 Administration Guide
The concepts of syslog-ng54
l Listspecificmacrostoincludeusingthekey()parameter.
l Definenewname-valuepairstoincludeusingthepair()parameter.
Theseparametersaredetailedinvalue-pairs().
Specifying data types in value-pairsBydefault,syslog-ngOSEhandleseverydataasstrings.However,certaindestinationsanddataformats(forexample,SQL,MongoDB,JSON,AMQP)supportothertypesofdataaswell,forexample,numbersordates.Thesyslog-ngOSEapplicationallowsyoutospecifythedatatypeintemplates(thisisalsocalledtype-hinting).Ifthedestinationdriversupportsdatatypes,itconvertstheincomingdatatothespecifieddatatype.Forexample,thisallowsyoutostoreintegernumbersasnumbersinMongoDB,insteadofstrings.
CAUTION:
Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error().
Tousetype-hinting,enclosethemacroortemplatecontainingthedatawiththetype:(""),forexample:int("$PID").
Currently the mongodb() destination and the format-json template function supportsdata types.
Example: Using type-hintingThefollowingexamplestorestheMESSAGE,PID,DATE,andPROGRAMfieldsofalogmessageinaMongoDBdatabase.TheDATEandPIDpartsarestoredasnumbersinsteadofstrings.
mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );
ThefollowingexampleformatsthesamefieldsintoJSON.
$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$M