AD104: DXL Re-invigoratedDick Annicchiarico | Product DeveloperAndre Guirard | Product Developer

2

Financial Disclaimer

The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion

3

Agenda● Progress in 2009 and future plans

● New automated testing tool for DXL round-trip fidelity

● Demos

4

Highlights of Progress in 2009● Overall mission: Round-trip fidelity of design elements● High-priority issues were addressed in 8.51● Set priorities and made plans for remaining work● Improved richtext fidelity● Designed a spiffy new testing tool● Began the drive towards full DXL fidelity

DXL

5

Progress in 8.51● Agent improvements, including full support for search expressions● Testing of low-level <note> format – verified as loss-less *

▬ Low-level format -- ForceNoteFormat=True and RichTextOption=RICHTEXTOPTION_RAW▬ * One fix in 8.52 for $Formula item on views and folders

● 34 SPRs plus additional issues fixed● Fixed reproducible hangs and crashes● Fixed 19 different errors (Exporter/Importer logs) from round-tripping ND

templates● Crowded Wisdom survey completed

6

Crowded Wisdom SurveyPriority Issue Weight

1 Don't crash 1792 Full implementation for view/folder design elements 1483 Full implementation of Agents 1344 Full implementation of Script Libraries 1295 Full implementation of Forms/Pages/Subforms 1286 Rich text: File attachment issues 1217 Make XPages and other text-based file elements optionally

include text rather than Base64102

8 Make all non-text based file elements give original binary file contents, rather than binary with CD record headers in it

100

9 Full implementation of Web Service Consumers 9310 Other rich text issues besides layout regions and attachments 9011 Full implementation of Web Service Providers 8612 Full implementation of Outlines 7913 Full implementation of Images 7114 Full implementation of Shared Fields 69

... 35 DXL exporter output must be valid to DTD and Schema 1

7

Progress for Future Releases● Specs for XPages and other file-based design elements● DXL Fidelity Verification Tool (DFVT)● <compositedata> - eliminates dropped richtext CD records● Fidelity retention processes● Nearing full round-trip fidelity of views and folders, and all they contain● 82 SPRs fixed (mostly for view/folder fidelity)

The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

8

XPages Specification - Highlights● Applies to File Resources, Stylesheets, Image Resources, Themes,

XPages, Custom Controls, Wiring Properties, Composite Applications, Components – all file-based design elements

● Current support (File, Stylesheet, Image) always emits Base64● New Exporter property FileFormat (RAW, ADAPTIVE, TEXT)

▬ ADAPTIVE (default) = raw if might not contain text, else text● Text formatting issues

▬ Character encoding issues – identifying character set, translations▬ Byte order marks▬ Line terminators

● Text data is in CDATA section(s), so we can retain it verbatim (especially important for XML files)

▬ If data itself contains CDATA termination sequence ]]> then data is split into multiple CDATA sections

9

DXL Fidelity Verification Tool (DFVT)● New comprehensive testing tool● It's an IBM Lotus® Domino® app!● 10,000+ lines of LotusScript® code● Allows us to verify full fidelity● More in just a few moments

10

DXL <compositedata> Element● Problem: Exporter doesn't recognize some richtext CD records, so their

data is not represented in DXL, and is lost in a round-trip● Ultimate solution: All CD records/fields/bits are represented in DXL● Upcoming solution: <compositedata> - save/restore CD record bits

▬ Holds raw canonical data for the CD record, encoded in Base64▬ Has contextual attributes that allow Importer to place record correctly in CD stream – proximity to

BEGIN and HOTSPOTBEGIN records, previous and next record signatures▬ Allows DXL to round-trip data that previously was lost, e.g. Layout regions▬ Not a complete solution – no help for records that are already handled, but only partially

richtextimproved

11

Fidelity Retention Processes● Must actively update DXL to remain at full fidelity as ND changes● We've implemented some internal processes to help us determine when

DXL changes are needed● We regularly round-trip all ND templates to detect regressions

immediately● We are considering additional ways to improve on this

12

View and Folder Full Fidelity● We are close to having DXL export and import of views and folders with

no data loss● Q: Is this bit-for-bit binary compatibility? A: No – but don't fret...

▬ It is binary compatibility whenever we can▬ We are documenting and will share all “acceptable differences” allowed by DFVT▬ The goal is functional equivalence!

● We've worked through lots of issues▬ View and column properties are very complex▬ Legacy design elements may differ from what current Designer produces▬ Lots of common constructs needed work as well

views folders Upcoming Release

13

Dependencies to Achieve View/Folder Fidelity● References to Shared Columns● Action Bars, Actions● References to Shared Actions● Named element references (images, etc.)● Formulas● LotusScript code● JavaScript code● Borders and Backgrounds● Colors● Fonts● and more!

14

Future Plans – 2010 and Beyond● Design element fidelity, continued

▬ Agents, Script Libraries, Shared Columns, Shared Actions should be relatively easy▬ Forms, Pages, Subforms

▬ will benefit from common constructs already done▬ richtext will be a large effort – may defer parts of it until rest of design element work is done

▬ XPages, and all other file-based design elements▬ Straightforward, since they are just a file and some metadata

● Designer source control integration● Other possibilities

▬ e.g. merging template customizations into an updated template

agents forms XPages. . .

The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

15

DFVT Mission● Provide automated DXL fidelity testing of ND apps at a binary level● Easy to use● Highly configurable● Transparent● Future possibility: Share with Design Partners

16

DFVT Summary● Does DXL high-level round-trip to get “before” and “after” design

elements● Exports “before” and “after” design elements using low-level DXL● Parses low-level DXL and generates detailed dump (kind of like

Notespeek)● Factors out acceptable differences (there are lots!)

▬ applies configurable rules▬ Success = no differences

● Guiding principle: Current Designer behavior is the “Gold”

17

DFVT Block Diagram

DXL Export DXL Import

DXL Export DXL Export

Parse and Apply Rules

Diff Tool

Parse and Apply Rules

Input Db

Round- trip Db

High-level DXL

Raw “Before”

DXL

Raw “After”

DXL

“Before” Output

“After” Output

DiffsGoal is no diffs!

18

DFVT Demo

19

DFVT Acceptable Differences● Lots of legitimate reasons for diffs, for example

▬ Timestamps▬ “Don't care” fill bytes to align records on 2-byte boundaries▬ Note references within the database – UNIDs differ▬ Font tables (same fonts, different order)▬ Formulas (local variable timestamps, stale column names)▬ Sometimes: data that is grayed out or hidden in the Designer UI▬ New Designer options added after design element originally saved, using previously unused fields

(e.g. original contains 0, round-trip contains new default value 4)● DFVT “factors out” acceptable diffs, leaves remaining diffs in output

This was created on 01/20/2010

This was created on xx/xx/xxxx

20

DXL Demos● Demo 1 – Form Letters● Demo 2 – Scanning for Design Errors● Demo 3 – Template Internationalization● Demo 4 – Action Bar UI Consistency

21

Demo 1 – Form Letters● Issue: Need to generate form letters with rich content

● Solution▬ Template document contains the fixed content with placeholders for variable content▬ A set of documents contain the variable content▬ Use DXL and DOM to do a mail merge

Dear ____,

22

Demo 1 Block Diagram

MailMerge process

Dear <name>,We hear you are crazy. Please come to our <clinic> clinic to have your head examined on <apptdate> at <appttime>.Signed,<signature>

Appt RecordName: Rex StinsonClinic: Maple RoadDate: 30 Jan 2010Time: 4:00 PM

DXL Exporter

DOM Parser

Dear Rex Stinson,We hear you are crazy. Please come to our Maple Road clinic to have your head examined on 30 Jan 2010 at 4:00 PM.Signed,

Wallace Bean, DirectorDXL

Importer

Name: Wallace BeanSignature:

Wallace Bean, Director

DXL Exporter

DOM Parser

Template document

23

Demo 1

Dear ____,

24

Demo 2 – Scanning for Design Errors● Issue: Template databases that don't fully adhere to conventions

▬ Hard to manually find all the places that need addressing● Solution:

▬ Export to DXL▬ Scan the DXL for any problems▬ Generate report

.ntf Generate Report

25

Demo 2 Block DiagramScan for missing field help:Server:Filepath:

Casbahdent\abc.nsf

Go

abc.nsf

DXL Exporter

SAX Parser Missing field help:Form A: field Age field Zip

26

Demo 2

.ntf Generate Report

27

Demo 3 – Template Internationalization● Issue: Template databases need to be localized to other languages

▬ Time consuming and error-prone to find all occurrences of strings needing translation▬ Changes need to be made in Designer, but translator might not be familiar with it

● Solution▬ Use DXL and XSLT style sheet to pull out all strings needing translation to a simple file▬ Translator makes changes to the simple file▬ Use DXL and XML processors to make changes to the corresponding design elements

English Español

28

Demo 3 Block Diagram

DXL Export XSLTEnglish Db

DXL

XSLT Style Sheet

XMLfor

translation

Translated XML

DOM Parse DOM ParseDOM Tree DOM Tree

Serialize

TranslatedDXL

DXL Import Spanish Db

views

29

Demo 3

English Español

30

Demo 4 – Action Bar UI Consistency● Issue: Different colors and fonts used in application's action bars,

therefore application lacks a consistent “look and feel”● Solution

▬ Identify a target design element to use as a model▬ Export design elements of that type to DXL▬ Process the DXL to generate a ToDo list of changes needed to enforce consistency▬ Automatically apply the ToDo list

31

Demo 4 Block Diagram

Action Bar Scan:Server:Filepath:Model design element:

Casbahdent\abc.nsf

abc.nsf

ViewBy TitleBy DateMy DocsBy Date

Go

NotesNoteCollection

1

DXL Exporter

2

DOM Parser

3

Action Bar ToDo List:Form WtItem

- change background to 133, 240, 240- display button border always

Fix automatically

4

DOM Parser

57

DXL Importer

6

32

Demo 4

33

Summary● Soon, you will be able to use DXL confidently for any processing of

views and folders● We've shown you the tool that we're using to guarantee DXL fidelity● We will be driving the remaining design elements to full fidelity

views folders

agents forms XPages . . .

The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

34

Q & A

● Also, come see us in the Meet the Developers lab – Dolphin Asia 1 – 2● Download examples from openntf.org● http://www-10.lotus.com/ldd/ddwiki.nsf/dx/ls-design-programming.htm

Manipulating Design Elements with LotusScript (or JavaTM)

Andre Guirard Dick Annicchiarico

35

Legal Disclaimer© IBM Corporation 2009. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

All references to companies other than IBM refer to fictitious companies and are used for illustration purposes only.