Bash parameter transforamtion on empty array triggers unset variable.

[Andrew Neff]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-musl
Compiler: gcc
Compilation CFLAGS: -g -O2 -Wno-parentheses -Wno-format-security
uname output: Linux 28e237a5e16f 5.5.7-200.fc31.x86_64 #1 SMP Fri Feb 28
17:18:37 UTC 2020 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-musl

Bash Version: 5.1
Patch Level: 0
Release Status: alpha

Description:
I do not know if this is related to bash 5.1 erroneously being
"a little aggressive about skipping over empty strings" mentioned
in "Declaring arrays with empty string in one line is bugged", but using
parameter transformation on an empty array, throws an error if "set -u" is
turned on. This was not how bash 4.4 and 5.0 worked. This bug makes it
impossible to check if an empty variable is an array using parameter
transformation while "set -u" is turned on

Repeat-By:
# Indirection
docker run -it --rm bash:5.1-alpha bash -uc 'foo(){ echo "${!***@a}";
}; bar=(); foo bar'
# Empty array
docker run -it --rm bash:5.1-alpha bash -uc 'bar=(); echo "${***@a}"'

# Declared unset array
docker run -it --rm bash:5.1-alpha bash -uc 'declare -a bar; echo
"${***@a}"'
# Empty associative array
docker run -it --rm bash:5.1-alpha bash -uc 'declare -A Bar=();
echo "${***@a}"'
# Declared unset associative array
docker run -it --rm bash:5.1-alpha bash -uc 'declare -A Bar; echo
"${***@a}"'

# I also tested on bash:devel-20200805, with the same results

Fix:
# All five of the above examples work in bash 4.4 and 5.0, and I
would expect the same behavior in bash 5.1
# Expected behavior that is already there in 5.1 alpha
docker run -it --rm bash:5.1-alpha bash -uc 'bar=(); echo
"${bar[@]}' # succeeds, but this is already consistent with bash 4.4 and
5.0. So this is expected to succeed.
docker run -it --rm bash:5.1-alpha bash -uc 'bar=(); echo
"${bar[@]+set}' # echos nothing, but this is already consistent with bash
3.2 and 5.0. So this is expected behavior
docker run -it --rm bash:5.1-alpha bash -uc 'bar=(); echo "${bar}'
# fails, but this is already consistent with bash 4.4 and 5.0. So this is
expected to fail, there is no 0th element to the array.
docker run -it --rm bash:5.1-alpha bash -uc 'bar=(); echo
"${bar+set}' # echos nothing, but this is already consistent with bash 3.2
and 5.0. So this is expected behavior

Thanks

[Chet Ramey]

Post by Andrew Neff
Bash Version: 5.1
Patch Level: 0
Release Status: alpha
        I do not know if this is related to bash 5.1 erroneously being
"a little aggressive about skipping over empty strings" mentioned
in "Declaring arrays with empty string in one line is bugged", but using
parameter transformation on an empty array, throws an error if "set -u" is
turned on. This was not how bash 4.4 and 5.0 worked. This bug makes it
impossible to check if an empty variable is an array using parameter
transformation while "set -u" is turned on

This was a bug in bash-5.0 (and 4.4) and was fixed in early March 2019 as
the result of

https://lists.gnu.org/archive/html/bug-bash/2019-03/msg00010.html

It's in CHANGES.

There are some other changes in how bash displays attributes of unset
variables when `nounset' is not enabled, but unset variables used in word
expansions should trigger an error -- with the usual @/* exceptions -- when
set -u is enabled.

[Andrew Neff]

Ah, I see the confusion.

The issue you pointed out, "@Q breaks set -o nounset" refers to quote
parameter expansion, as does the line in CHANGES-5.1, 1.q, which makes
sense to call this a bug that was allowed in bash 4.4 and 5.0.

I should have specified, the focus of this issue is the "@a" expansion. It
makes sense that @Q/E/P/A expansion should not work on unset variables with
nounset enabled. However, @a is uniquely different, in that it does not
have to do with the value of the variable, but rather the variable type. My
"set -eu" compatible library uses the @a expansion (for bash 4.4 and newer)
because it had previously worked on unset values with set -u enabled, which
was a very useful feature.

Here are 3 specific details I would like to address:

1. @a expansion should work on unset variables with "set -u" in bash 5.1.
It seems like the correct thing to do. Only @a expansion. This has been a
very useful feature in bash 4.4 and 5.0.
Should fail: (set -eu; declare -a x; echo "${***@Q}")
Should not fail: (set -eu; declare -a x; echo "${***@a}")

2. With "set -u", the following works in bash 4.4 or newer (and makes sense
that it works): (set -eu; x=(); echo "${x[@]}")
Here x is not unset, it is set to an empty array. This expansion make sense
with nounset turned on because x is not unset, it is set to ()
However, this fails: (set -eu; x=(); echo "${***@a}")
This is an inconsistent behavior, and it seems to me the ${***@a} should
definitely work here, with nounset turns on

3. The same as #2, but for associative arrays
Works: (set -eu; declare -A x=(); echo "${x[@]}")
Does not work, but should: (set -eu; declare -A x=(); echo "${***@a}")

Thanks

Post by Chet Ramey

Post by Andrew Neff
Bash Version: 5.1
Patch Level: 0
Release Status: alpha
I do not know if this is related to bash 5.1 erroneously being
"a little aggressive about skipping over empty strings" mentioned
in "Declaring arrays with empty string in one line is bugged", but using
parameter transformation on an empty array, throws an error if "set -u"

is

Post by Andrew Neff
turned on. This was not how bash 4.4 and 5.0 worked. This bug makes it
impossible to check if an empty variable is an array using parameter
transformation while "set -u" is turned on

This was a bug in bash-5.0 (and 4.4) and was fixed in early March 2019 as
the result of
https://lists.gnu.org/archive/html/bug-bash/2019-03/msg00010.html
It's in CHANGES.
There are some other changes in how bash displays attributes of unset
variables when `nounset' is not enabled, but unset variables used in word
set -u is enabled.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates

[Chet Ramey]

Post by Andrew Neff
Ah, I see the confusion.
parameter expansion, as does the line in CHANGES-5.1, 1.q, which makes
sense to call this a bug that was allowed in bash 4.4 and 5.0.

Not quite. The report revealed that none of the parameter transformations
obeyed `set -u'. The fix was to make that happen, so that parameter
transformations were aligned with the other word expansions.
Sure, you want an exception.

Post by Andrew Neff
It
have to do with the value of the variable, but rather the variable type.

This is why it works on unset variables when `set -u' isn't enabled. The
question is whether that makes it special enough to be an exception.

Post by Andrew Neff
very useful feature in bash 4.4 and 5.0.

OK, you found it useful in something you wanted to do. You're restating the
proposition.

Post by Andrew Neff
2. With "set -u", the following works in bash 4.4 or newer (and makes sense
Here x is not unset, it is set to an empty array. This expansion make sense
with nounset turned on because x is not unset, it is set to ()

That's not why. The variable is indeed unset. The reason it doesn't fail
is because there's a special POSIX carveout for $@ with no positional
parameters and I extended it to ${var[@]}. The expansions of $@ and ${A[@]}
parallel each other in just about every other way, so it made sense.

Post by Andrew Neff
3. The same as #2, but for associative arrays

The same carveout for ${A[@]}.

So I guess the question is whether or not @A and @a are exceptional enough.