This is not a bug. The method StringTokenizer.hasMoreElements() had an
unexpected side-effect of consuming the previous delimiters that was not
speced. The fix for bug 4238266 corrected this. So the behavior of the
following code before that fix was,
   StringTokenizer st = new StringTokenizer("ab", "b");
   System.out.println(st.nextToken());
   st.hasMoreElements();
   System.out.println(st.nextToken("");
   

generates,
   a
followed by an exception.
 
would be different form the behavior of,
   StringTokenizer st = new StringTokenizer("ab", "b");
   System.out.println(st.nextToken();
   System.out.println(st.nextToken("");
which generates,
a
b

 
Hence the fix.  




java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)

The StringTokenizer class seems to function differently in the latest release
of JDK1.3.  In this new release, changing the delimiter from the initial value
causes the next call to nextToken() to include the previous delimiter.

<pre>

import java.util.*;
class Test
{
  public static void main(String[] args)
  {
    StringTokenizer st = new StringTokenizer("test=abcd efg hij", "=");
    System.out.println(st.nextToken());
    while (st.hasMoreTokens())
    {
      System.out.println(st.nextToken(" "));
    }
  }
}

Output with latest JDK1.3
test
=abcd
efg
hij

Output before lastest JDK1.3
test
abcd
efg
hij

</pre>
(Review ID: 104907) 
======================================================================

Post processing the string to check for the previous delimiter.
======================================================================

This is not a bug. The method StringTokenizer.hasMoreElements() had an
unexpected side-effect of consuming the previous delimiters that was not
speced. The fix for bug 4238266 corrected this. So the behavior of the
following code before that fix was,
   StringTokenizer st = new StringTokenizer("ab", "b");
   System.out.println(st.nextToken());
   st.hasMoreElements();
   System.out.println(st.nextToken("");
   

generates,
   a
followed by an exception.
 
would be different form the behavior of,
   StringTokenizer st = new StringTokenizer("ab", "b");
   System.out.println(st.nextToken();
   System.out.println(st.nextToken("");
which generates,
a
b

 
Hence the fix.  

 xxxxx@xxxxx  2000-05-22

How is this not a bug?!  It changes the base functionality of the class without documenting that it was 
done, in the enhancements documentation.  This class should work exactly as it always has

I agree that this is a bug, but in my case, I got around the problem by creating the tokenizer with
returnTokens set to true and explicitly skipping the tokens.  This provided the desired results.

Sample code

               StringTokenizer tokens=new StringTokenizer(fileLogValue,",", true);
                String newLogClassName = tokens.nextToken(",").trim();
                String ignore=tokens.nextToken(",");
                String newLogLevel = tokens.nextToken(",").trim();
                ignore=tokens.nextToken(",");
                String newLogAddress = tokens.nextToken("").trim(); // change of token
                /* the last nextToken was returning the previous comma */
                /* using the above method, skips the comma */

This is a bug and it should be fixed. It is a serious incompatibility with previous Java 2 JDK's and JDK 1.x 
JDK's.

Of course it's a bug !
I cannot anderstand how it could go through the non-regression tests.
It prevented us to use the JDK1.3 until we suppressed all access to this class.

Sure looks like a bug to me. I could certainly break alot of
peoples code.

It sure breaks a lot of code, including third-party java 
code, which we have no control over.

For example the excellent servlet engine JRun 2.x. The 
handling of cookies here is obviously dependent on a 
StringTokenizer, and when JRun is used with JDK 1.3, all 
cookie values are returned with "="+<real value>, an equal 
sign is added in front of every cookie value. 

That means a LOT of servlets must be changed...

If this is not a bug, bite me!

What comments is the "evaluation" referring to?  Nowhere does anyone attempt to explain how this is "Not a  
bug".  If changing the behavior was intentional, it should have been documented and an alternative 
constructor should have been introduced to obtain the new behavior.  Changing the behavior without doing 
so is a bug.  I can work around it in my code, but I can't easily work around it in class libraries that I 
depend on.  If the dates on this bug report are correct, the bug was reported early and closed a week later!  
The s**t is going to start hitting the fan as people try to pick up JDK1.3 and really using it (supposedly now 
that the bugs have been worked out in beta!).

4338282 and 4238266 are mirror opposites.  Recommend rolling
back fix 4238266, which would resolve the very real
incompatibility issue.  Need to rollback the JCK test case
too.

 -Thomas Wang

i'm sorry; this is definately a bug;
it makes nearly all of my usage of StringTokenizer heretofore
totally obsolete and in need of rewriting;

This is definitely a bug.  I spent a lot of time trying to 
figure out why our code stopped working when running 
1.3.1.  You should have documented this and provided an 
alternate contructor.  I have lots of code to fix now!!

This one never bothered me - I never change my delmiter.

Workaround: Post processing the string to check for the previous delimiter.

Please tell me there is a method to get the previous delimiter in StringTokenizer class...