Travis CI: doctest failure

[vstinner]

BPO	36597
Nosy	@vstinner, @methane, @ambv, @JulienPalard
PRs	bpo-36597: Disable Travis CI doctest job #12775 bpo-36597: fix random doctest failure #12776 [3.7] bpo-36597: fix random doctest failure #12778 bpo-36597: fix weakref example code #12779

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-11.10:38:52.476>
created_at = <Date 2019-04-11.02:49:25.530>
labels = ['release-blocker', '3.8', 'docs']
title = 'Travis CI: doctest failure'
updated_at = <Date 2019-04-11.10:38:52.476>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-04-11.10:38:52.476>
actor = 'methane'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-04-11.10:38:52.476>
closer = 'methane'
components = ['Documentation']
creation = <Date 2019-04-11.02:49:25.530>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36597
keywords = ['patch']
message_count = 18.0
messages = ['339919', '339932', '339933', '339934', '339935', '339936', '339937', '339938', '339939', '339940', '339941', '339942', '339943', '339944', '339945', '339946', '339951', '339957']
nosy_count = 5.0
nosy_names = ['vstinner', 'methane', 'docs@python', 'lukasz.langa', 'mdk']
pr_nums = ['12775', '12776', '12778', '12779']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36597'
versions = ['Python 3.8']

[vstinner]

On my #12770 the doctest job of Travis CI failed with:

https://travis-ci.org/python/cpython/jobs/518572326

mkdir -p build
Building NEWS from Misc/NEWS.d with blurb
PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -D latex_elements.papersize= -q -W -j4 -W . build/doctest
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
/home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead.
env.note_versionchange('deprecated', version[0], node, self.lineno)
Warning, treated as error:

---

File "library/unittest.mock-examples.rst", line ?, in default
Failed example:
m.one().two().three()
Expected:
<MagicMock name='mock.one().two().three()' id='...'>
Got:
obj dead or exiting
<MagicMock name='mock.one().two().three()' id='140222049958880'>
Makefile:44: recipe for target 'build' failed
make[1]: *** [build] Error 2

--

I can reproduce this issue on Linux with:

$ cd Doc
$ make venv
$ PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -D latex_elements.papersize= -q -W -j4 -W . build/doctest 

I get random errors:

Warning, treated as error:

---

File "library/datetime.rst", line 686, in default
Failed example:
d.strftime("%A %d. %B %Y")
Expected:
'Monday 11. March 2002'
Got:
'lundi 11. mars 2002'

Warning, treated as error:

---

File "library/collections.rst", line 914, in default
Failed example:
p._asdict()
Expected:
{'x': 11, 'y': 22}
Got:
OrderedDict([('x', 11), ('y', 22)])

Warning, treated as error:

---

File "library/unittest.mock.rst", line ?, in default
Failed example:
mock.call_args.args
Expected:
(3, 4)
Got:
args

The virtual environment uses Sphinx 2.0.1. Can it be a change in Sphinx 2 default configuration?

[vstinner]

Well, same problem with Sphinx 1.8.5. No idea why the job started to fail.

[methane]

On Thu, Apr 11, 2019 at 11:49 AM STINNER Victor <report@bugs.python.org> wrote:

File "library/unittest.mock-examples.rst", line ?, in default
Failed example:
m.one().two().three()
Expected:
<MagicMock name='mock.one().two().three()' id='...'>
Got:
obj dead or exiting
<MagicMock name='mock.one().two().three()' id='140222049958880'>

I'm not sure where "obj is dead or exiting" came from.

File "library/datetime.rst", line 686, in default
Failed example:
d.strftime("%A %d. %B %Y")
Expected:
'Monday 11. March 2002'
Got:
'lundi 11. mars 2002'

It seems it is caused by non English locale.

File "library/collections.rst", line 914, in default
Failed example:
p._asdict()
Expected:
{'x': 11, 'y': 22}
Got:
OrderedDict([('x', 11), ('y', 22)])

You may run doctest with existing Python, not Python 3.8.

File "library/unittest.mock.rst", line ?, in default
Failed example:
mock.call_args.args
Expected:
(3, 4)
Got:
args

I'm not sure about this.

Inada Naoki <songofacandy@gmail.com>

[vstinner]

It seems it is caused by non English locale.

Right, but why did it start to fail since yesterday?

You may run doctest with existing Python, not Python 3.8.

I used the command from the .travis.yml job: "... make -C Doc/ PYTHON=../python "...". It explicitly picks the Python it just compiled.

Why would we like to check the doctests using a random Python version, whereas it's Python 3.8 documentation?

[vstinner]

8 hours ago, the job still passed at the top of the the master branch (commit a9bd892):

https://travis-ci.org/python/cpython/jobs/518281237

"Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 Pygments-2.3.1 Sphinx-2.0.1 alabaster-0.7.12 babel-2.6.0 blurb-1.0.7 certifi-2019.3.9 chardet-3.0.4 docutils-0.14 idna-2.8 imagesize-1.1.0 packaging-19.0 pyparsing-2.4.0 python-docs-theme-2018.7 pytz-2019.1 requests-2.21.0 six-1.12.0 snowballstemmer-1.2.1 sphinxcontrib-applehelp-1.0.1 sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.1 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 urllib3-1.24.1"
...
PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -D latex_elements.papersize= -q -W -j4 -W . build/doctest
...
The command "xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest" exited with 0.

---

6 hours ago, it failed:

https://travis-ci.org/python/cpython/jobs/518572326

Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 Pygments-2.3.1 Sphinx-2.0.1 alabaster-0.7.12 babel-2.6.0 blurb-1.0.7 certifi-2019.3.9 chardet-3.0.4 docutils-0.14 idna-2.8 imagesize-1.1.0 packaging-19.0 pyparsing-2.4.0 python-docs-theme-2018.7 pytz-2019.1 requests-2.21.0 six-1.12.0 snowballstemmer-1.2.1 sphinxcontrib-applehelp-1.0.1 sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.1 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 urllib3-1.24.1

Warning, treated as error: (...)

--

In the 2 builds, pip installed the the same Python modules versions.

I compared the raw logs of the 2 builds. In short, they are the same.

Maybe the job already failed previously, but only randomly?

[methane]

> It seems it is caused by non English locale.

Right, but why did it start to fail since yesterday?

Is it an error start to fail since yesterday?
As far as reading your first report, this is happen only in your local
environment.

> You may run doctest with existing Python, not Python 3.8.

I used the command from the .travis.yml job: "... make -C Doc/ PYTHON=../python "...". It explicitly picks the Python it just compiled.

This error is in your environment too. Maybe you re-used old venv?

Why would we like to check the doctests using a random Python version, whereas it's Python 3.8 documentation?

Because it "doctest". Python behavior (repr) can be changed.
Python 3.8 replaces some OrderedDict usage with regular dict.

[vstinner]

New changeset adff01e by Victor Stinner in branch 'master':
bpo-36597: Disable Travis CI doctest job (GH-12775)
adff01e

[vstinner]

I mark this issue as a release blocker to not forget it. I disabled the Travis CI doctest job to repair the CI, but we have to investigate why it started to fail yesterday.

[vstinner]

As far as reading your first report, this is happen only in your local
environment.

See the beginning of my first message, the doctest prevents me to merge a PR. But I tested on the master branch without my change, and I get the same issue locally, and the issues seems to be unrelated to my change.

"""
On my #12770 the doctest job of Travis CI failed with:

https://travis-ci.org/python/cpython/jobs/518572326
"""

[tirkarthi]

"Object dead or exiting" is a less frequent random error with doctest. I am not sure from where it occurs but occurs for different tests at different times and is overlooked in Travis since the next runs are fine.

Sample old builds :

https://travis-ci.org/python/cpython/jobs/517606752
https://travis-ci.org/python/cpython/jobs/510847596

[methane]

See the beginning of my first message, the doctest prevents me to merge a PR.

You did not show travis build link which failed other than "obj dead or exiting" error.
Don't all other errors happened only on your environment? If no, please link to travis build link.

As far as I can confirm now, only "obj dead or exiting" error is problem in Travis.

But I tested on the master branch without my change, and I get the same issue locally, and the issues seems to be unrelated to my change.

I suspect your environment (locale and venv used to run doctest), not your changes.

"""
On my #12770 the doctest job of Travis CI failed with:

https://travis-ci.org/python/cpython/jobs/518572326
"""

Only "obj dead or exiting" happened in the job.