Easy contributable internationalization process with Sphinx @ pyconsg2015

Easy contributable internationalization process with Sphinx

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp 1

Who am I

@shimizukawa

1. Sphinx co-maintainer

2. Sphinx-users.jp

3. PyCon JP committee

BePROUD co, ltd.2

http://pycon.jp/

3

Registration: Opened

Speaker Call for Proposals: until July 15th.

4

Agenda

1. Sphinx introduction & Setup for i18n

2. A easy way and Non easy ways to translate

3. Sphinx i18n feature

4. Automated translation process with several services

5. Tips, tricks, traps

5

Question

6

Question1

How many people do you use Open Source

Software?

7

Question2

How many people may have contributed to the

OSS?

8

Question3

Does the translation into familiar language

contribute to other developers?

9

10

Translator

Product Author

Translated Docs

Readers

English Docs

Translators

Tools & Services

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

11

1. Sphinx introduction& Setup for i18n

12

What is Sphinx?

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

13


Sphinx

reSTreSTreStructuredTe

xt(reST)

reST Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

The history of Sphinx (short ver.)

14


The father of Sphinx

Too hard to maintenance

~2007

Easy to writeEasy to

maintenance2007~

Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

15


Relation between Sphinx and Docutils

Sphinx based upon Docutils library

Docutils/Sphinx supports reStructuredText it called reST, it's an extensible markup(PEP287 / accepted at 2002)

16


Sphinx

reST Parser

HTML Builder

ePub Builder

LaTeX Builder

HTMLtheme

docutils

Comparing Docutils and Sphinx

Docutils Sphinx

1. Single page

2. Link to others with full name

3. Cross reference in a page

4. Writers: html, latex, man, xml, ...

5. Standard reST markup specs

1. Multiple Pages

2. toctree to connect all pages under single tree structure

3. Cross reference over each other pages

4. Additional writers: html(w/ themes), pdf(latex), texinfo, epub, …

5. Additional markup specs:autodoc directive and so on

17


SPHINX

docutils

HTML Builder HTML theme(Jinja2)

gettext Builder

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx i18n feature (built-in)

18


reST Parser(directive / role)

doctree(Intermediat

e)

reSTreSTreStructuredText(reST)

$ pip install sphinx

Support tools for translation

How to install Sphinx with i18n tools

To build a sphinx document

$ pip install sphinx-intl$ pip install transifex-client=0.8

19


$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

20


2. Easy contributable internationalization process with Sphinx

22

What is internationalization?

23


I18N

What is easy contributable?

24


Not a easy way (example 1/3)

The manual are provided only in the HTML files

Rewriting reST files

25



The manual are generated fromreST files and

docstrings in the source files

Rewriting reST filesRewriting docstrins in the source files.

26



OK, we are engineers. We can use git!

Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to

do!)git commitgit pullSometimes resolve conflictgit push

27


Not easy vs easy

Not a easy way Easy way

1. Learn git2. Learn GitHub3. git clone to get the

code4. Translation5. git commit, pull,

push6. Resolve conflict7. Build html your self

1. No git2. No Github3. No file4. Translation5. Update

Automatically6. No conflict7. You can get a HTML

output w/o hand-build.

28



29

Two i18n features of Sphinx

Output pot files:from reST

Input po files:to generate translated HTML

30


reST

pot

reST

html

po

Translation flow

Generate pot

Translate pot into po

Generate Translated HTML

31


reST

pot

reST

html

po

pot po

Translator

Pleasetranslate

Translate!Thanks for yourContribution!

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""

msgid "For example:"msgstr ""

32

Output pot files 3. Sphinx i18n feature

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ ls _build/gettextapi.pot examples.pot generated.pot index.pot

generated.pot

reST

pot

34

Preparing po files to translate

$ sphinx-intl update -p _build/gettext -l zh_cnCreate: locale/zh_cn/LC_MESSAGES/api.poCreate: locale/zh_cn/LC_MESSAGES/examples.poCreate: locale/zh_cn/LC_MESSAGES/generated.poCreate: locale/zh_cn/LC_MESSAGES/index.po

At first, sphinx-intl copy pot files and rename them

pot po

sphinx-intl

$ make gettext$ sphinx-intl update -p _build/gettext -l zh_cnNot Changed: locale/zh_cn/LC_MESSAGES/api.poUpdated: locale/zh_cn/LC_MESSAGES/examples.po +3, -1Not Changed: locale/ma/LC_MESSAGES/generated.po

After change the document, sphinx-intl update differences


Translate po files

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr ""

generated.po

pot po

sphinx-intl

Translator

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr " 序列 `òbj`` 以 JSON 格式 :class:`str` 。 "

generated.po

Translate by using Vim, Emacs, OmegaT, ...

35


Input po files

36


reST

html

po

Translated

$ make html...Build finished. The HTML pages are in _build/html.

Big picture

37


reST

pot

html

po

make gettext

sphinx-intl

Translator

make html

Doc Author

TranslationCatalog

Translatedcatalog

Demo

38


39

Entire process to translate sphinx docs

40


reST

pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

Translator

Translator

Translator

Autor / Translator

upload

Translator

clone

Translator

To Be

41


reST

pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

upload

Translators

To BeAutomated

To BeParallel

clone

Translation tool types

Vim / Emacs (Editors)Edit local filesTranslation support plugins are available.

OmegaT (Translation Tools)Edit local files.Translation support features.

Transifex (Services)Edit onlineTranslation support

features.

42


po

Translators

To BeParallel

Be Parallel by using Transifex

Transifex provides...As API:Upload potDownload po

As Web console:GlossaryTranslation memoryRecommendationAuto-translation

43


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-client

Download

Translation on Transifex web console

44


Original Text

Translated Text(you should keep reST syntax)

Suggestions fromTranslation Memory (TM)

Original(pot)

Translation(po)

Copy orig to translation

Machine translation

Save

Review (if needed)

Translators

Parallel

1

2

4

3

5

6

To Be Automated

45


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-client

Download

reST

html

make gettext

make html

Doc Author

upload

To BeAutomated

clone

To Be Automated

46


pot

Upload sphinx-intltransifex-client

po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

The procedure for automation

1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html

47


pot


po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomate

d

1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources

# update .tx/config8. tx push -s # push pot files to

transifex9. tx pull -l zh_cn # pull po files from

transifex10. make html SPHINXOPTS="-D language=ma"

Shell commands for automation

run.sh

$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh

48


The drone.io

49

WebHook

Deploy

Clone repositoryRun shell script

The drone.io is a continuous integration service.


GitHub + drone.io + S3

50

GitHub

Amazon S3

Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html


2

1

3

Be Automated

51

pot


po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


Automated by drone.io

52

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


View from doc author

Doc Author doesn't require annoying procedure.

UpdateTranslation Source

Doc Author

Notify

See

Commit

53


View from doc translators

No gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-

build.

Translators

Parallel

See

TranslateTranslated Pages

54


The entire automated process

55

UpdateTranslation Source

Doc Author

Translators

Parallel

Notify

See

See

Publish

Translate

Commit

DownloadTranslations

Notify

Automated


Summary

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

56



57

TIP: Drone.io limits 15mins for a build

Drone.io limits 15mins for a build.Install from wheel package may help you

58


1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -

2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh

ex. https://bitbucket.org/sphinxjp/docutils-translation/

TRAP: Version of transifex-client

transifex-client 0.11b3 is not stable (for me)Especially for Windows users

If you have encountered trouble with using newer version, please try:

59


$ pip install "transifex-client=0.8"

Drone.io only list-up repositories which you have admin permission.

Prepare a empty repository to create a drone.io project.

60

TRICK: Preparing Drone.io project5. Tips, tricks, traps

Examples

Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin

Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin

Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin

61


https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin








Questions?@shimizukawa

Grab me after the session :)

62

Thanks :)

63

Easy contributable internationalization process with Sphinx @ pyconsg2015

Technology

Transcript of Easy contributable internationalization process with Sphinx @ pyconsg2015