rustdoc crashes on nightly nightly-2019-08-22

[idubrov]

cargo doc crashes on nightly (nightly-2019-08-22) and also on stable (1.37.0) with "nightly" features turned on via RUSTC_BOOSTRAP=1. It does not trigger on stable without RUSTC_BOOTSTRAP=1 flag.

Here is the output I'm getting:

thread 'rustc' panicked at 'could not find markdown in source', src/libcore/option.rs:1166:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.39.0-nightly (e44fdf979 2019-08-21) running on x86_64-apple-darwin

error: Could not document `rustdoc-crash`.

Caused by:
  process didn't exit successfully: `rustdoc --edition=2018 --crate-name rustdoc_crash src/lib.rs -o /Users/idubrov/Projects/rustdoc-crash/target/doc --color always -L dependency=/Users/idubrov/Projects/rustdoc-crash/target/debug/deps --extern derive=/Users/idubrov/Projects/rustdoc-crash/target/debug/deps/libderive-aeb93296cea2c806.dylib` (exit code: 1)

I created a small repro case + README.md here: https://github.com/idubrov/rustdoc-crash (it only happens when there is a certain combination of procedural macros, so I cannot make it a single file).

[idubrov]

The source code is:

#[derive::first]
#[derive::second]
pub enum Boom {
    /// [Whatever]
    Bam,
}

with derive crate being a procedural macro crate with:

extern crate proc_macro;

#[proc_macro_attribute]
pub fn first(
  _attr: proc_macro::TokenStream,
  item: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
  item
}

#[proc_macro_attribute]
pub fn second(
  _attr: proc_macro::TokenStream,
  item: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
  // Note: if I return `item` as-is (same as `first` above), it resolves the issue!
  let mut out: proc_macro::TokenStream = proc_macro::TokenStream::new();
  out.extend(item);
  out
}

[idubrov]

Couple of things that are important here:

1. It only happens with two procedural macros.
2. Second macro has to do that TokenStream::extend.
3. Doc attribute has to be [Whatever], does not reproduce with anything else.

[nox]

Just a heads-up: AFAIK if a bug only occurs on stable with RUSTC_BOOTSTRAP=1 (I understand that it is not the case here), then it should probably be disregarded, because RUSTC_BOOTSTRAP=1 isn't supposed to be used when not bootstrapping the compiler.

[nox]

The bug itself reminds me of #46489.

[estebank]

@nox it happens on nightly as well. That detail was included to give context that evidently the ICE has been present for longer than 12 weeks, I believe.

[estebank]

With the following patch in the linked repo,

diff --git a/src/librustdoc/passes/mod.rs b/src/librustdoc/passes/mod.rs
index 641a6df221..9970ac840f 100644
--- a/src/librustdoc/passes/mod.rs
+++ b/src/librustdoc/passes/mod.rs
@@ -409,7 +409,10 @@ crate fn source_span_for_markdown_range(

     'outer: for (line_no, md_line) in md_lines.enumerate() {
         loop {
-            let source_line = src_lines.next().expect("could not find markdown in source");
+            let source_line = match src_lines.next() {
+                Some(line) => line,
+                None => break,
+            };
             match source_line.find(md_line) {
                 Some(offset) => {
                     if line_no == starting_line {

the output is not ideal, but preferable to the ICE:

 Documenting derive v0.1.0 (/Users/ekuber/workspace/rustdoc-crash/derive)
 Documenting rustdoc-crash v0.1.0 (/Users/ekuber/workspace/rustdoc-crash)
warning: `[Whatever]` cannot be resolved, ignoring it...
 --> src/lib.rs:1:2
  |
1 | #[derive::first]
  |  ^^^^^^^^ cannot be resolved, ignoring
  |
  = note: `#[warn(intra_doc_link_resolution_failure)]` on by default
  = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`

I'll take a look to see if we can somehow keep the span for the entire attr doc, or at worst pointing at the correct derive::second macro.

[estebank]

Changing the patch to be

diff --git a/src/librustdoc/passes/mod.rs b/src/librustdoc/passes/mod.rs
index 641a6df221..9970ac840f 100644
--- a/src/librustdoc/passes/mod.rs
+++ b/src/librustdoc/passes/mod.rs
@@ -409,7 +409,10 @@ crate fn source_span_for_markdown_range(

     'outer: for (line_no, md_line) in md_lines.enumerate() {
         loop {
-            let source_line = src_lines.next().expect("could not find markdown in source");
+            let source_line = src_lines.next()?;
             match source_line.find(md_line) {
                 Some(offset) => {
                     if line_no == starting_line {

gives us

warning: `[Whatever]` cannot be resolved, ignoring it...
  |
  = note: `#[warn(intra_doc_link_resolution_failure)]` on by default
  = note: the link appears in this line:

          [Whatever]
           ^^^^^^^^
  = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`

which makes more sense but is missing a primary span.

Edit: I see no way to recover the original span after its TokenStream has been modified. I guess we already rely on proc macro writers to be nice when it comes to spans, so this is acceptable as well.