TechVidvan

Java StringTokenizer Class with Example

by

We have already worked with Strings in our article of Strings and in the most recent article of StringBuffer in Java.

We always perform operations on Strings with the help of each character present in the String. What if we want to work with each word in a String or a sentence?

With String Tokenizer in Java, we can break a String into words, and such words are known as tokens.

In this article, we will discuss the StringTokenizer class in Java which accomplishes this task of breaking a String into tokens and performing operations on these tokens.

We will learn each method and constructor associated with this class with examples. So let’s begin.

StringTokenizer in Java

A StringTokenizer class is a class present in the java.util package and it is used to break a String into tokens.

In other words, we can split a sentence into its words and perform various operations like counting the number of tokens or breaking a sentence into tokens.

There are constructors and methods in this StringTokenizer that help us to split a sentence into tokens. StringTokenizer, tokenize the string on the basis of the delimiters provided to the String tokenizer class object.

The Common delimiters are the whitespaces, tab, newline, carriage return, and form feed.

These delimiters are taken as default and if a user wants to provide its own delimiter then he can provide by defining the delimiter in the parameter as an argument.

Let’s take a simple example to understand the use of StringTokenizer class in Java.

Example of String Tokenizer class in Java

Let’s now fall on the programming parameters of the StringTokenizer Class and take a complete eye of this.

Java StringTokenizer Constructors

StringTokenizer generally describes three types of constructors:

1. StringTokenizer(String str)

This constructor is implemented to perform tokenization of a particular string that is being provided in the parameter.

This constructor takes all the default delimiters which have been defined in the definition of the StringTokenizer class.

The delimiter is: WhiteSpaces, newline, tab, carriage return “\r”, Line feed “\n”, and form feed “\f”.

2. StringTokenizer(String str, String delimiter)

This constructor is implemented to perform string tokenization based on the delimiter provided by the user in the argument. This can be explained by an example:

StringTokenizer st = new StringTokenizer(“techvidvan, article, on, StringTokenizer”, “,”);
while (st.hasMoreTokens())
     System.out.println(st.nextToken());

Output:
techvidvan
article
on
StringTokenizer

3. StringTokenizer(String str, String delimiter, boolean flag)

This Constructor is implemented to perform string tokenization based on the delimiter and has additional functionality to display delimiter also.

This can be explained by an example:

Example of boolean = false :

StringTokenizer st = new StringTokenizer(“techvidvan, article, on, StringTokenizer”, “, ”, false);
while (st.hasMoreTokens())
    System.out.println(st.nextToken());

Here if the value of boolean is FALSE then token doesn’t contain the delimiter i.e output is: “techvidvan”, “ article”, “ on”, “ StringTokenizer”

Example of boolean = true :

StringTokenizer st = new StringTokenizer(“techvidvan, article, on, StringTokenizer”, “, ”, true);
while (st1.hasMoreTokens())
    System.out.println(st.nextToken());

Here if the value of boolean is TRUE then token contain the delimiter i.e.,

The output is:
techvidvan
,
article
,
on
,
StringTokenizer

Class Diagram of Java StringTokenizer class

Class Diagram of String Tokenizer in Java

Useful Methods of StringTokenizer class in Java

The StringTokenizer class contains a number of useful instance methods that we can use to isolate tokens.

Let’s start learning each method of the StringTokenizer class. To use these methods, we must obviously first create the object of the StringTokenizer class as shown below:

StringTokenizer st = new StringTokenizer(String);

Here, the argument that we passed is a String that we need to tokenize.

1. int countTokens()

The countToken() method returns the number of tokens that are separated by any white space in the given string.

Using this method we can know the number of tokens in the String, and we can, therefore, use this number of tokens as a loop parameter to process the whole string very easily.

Let’s discuss this method with an example:

Code to understand the countTokens() method of StringTokenizer class:

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class CountTokensMethod {
  public static void main(String[] args) throws IOException {
    String myString = "This is TechVidvan Tutorial of Java";
    StringTokenizer st = new StringTokenizer(myString);
    int numberOfTokens;
    numberOfTokens = st.countTokens();
    System.out.println("Input string is:" + myString);
    System.out.println("The number of tokens in the string is: " + numberOfTokens);
  }
}

Output:
Input string is: This is TechVidvan Tutorial of Java
The number of tokens in the string is: 6

2. String nextToken()

The nextToken() method of the StringTokenizer class returns the next token in the form of String. When we use it for the first time, it returns the next token as the first token of the string.

Code to understand the nextToken() method of StringTokenizer class:

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class NextTokenMethod {
  public static void main(String[] args) throws IOException {
    String myString = "This is TechVidvan Tutorial of Java";
    StringTokenizer st = new StringTokenizer(myString);
    System.out.println("Input string is: " + myString);
    while (st.hasMoreTokens()) {
      System.out.println("The Next token is: " + st.nextToken());
    }
  }
}

Output:
Input string is: This is TechVidvan Tutorial of Java
The Next token is: This
The Next token is: is
The Next token is: TechVidvan
The Next token is: Tutorial
The Next token is: of
The Next token is: Java

3. String nextToken(String delimiter)

The nextToken(String delimiter) method is the same as the nextToken() method that we discussed above, the only difference is that it returns the next token on the basis of the delimiter that we provide as an argument to this method.

We can use any delimiter such as a symbol or any number or any character.

Code to understand the nextToken(String delimiter) method of StringTokenizer class:

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class NextTokenMethod {
  public static void main(String[] args) throws IOException {
    String myString = "This$is$TechVidvan$Tutorial$of Java";
    StringTokenizer st = new StringTokenizer(myString);
    System.out.println("Input string is: " + myString);
    while (st.hasMoreTokens()) {
      System.out.println("The Next token with $ delimiter is: " + st.nextToken("$"));
    }
  }
}

Output:
Input string is: This$is$TechVidvan$Tutorial$of Java
The Next token with $ delimiter is: This
The Next token with $ delimiter is: is
The Next token with $ delimiter is: TechVidvan
The Next token with $ delimiter is: Tutorial
The Next token with $ delimiter is: of Java

4. boolean hasMoreTokens()

This method just checks whether there is any more token present in the String. It returns true if a token is available and false if no token is available.

We can use it in the while loop to process the whole string until no more token is available.

Code to understand the hasMoreTokens() method of StringTokenizer class:

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class HasMoreTokensMethod {
  public static void main(String[] args) throws IOException {
    String myString = "This is TechVidvan Tutorial of Java";
    StringTokenizer st = new StringTokenizer(myString);
    System.out.println("Input string is: " + myString);
    int n = st.countTokens();
    while (n != 0)
    if (st.hasMoreTokens()) {
      System.out.println("The token is present");
      n--;
    }
    System.out.println("There is no more token in the string");
  }
}

Output:
Input string is: This is TechVidvan Tutorial of Java
The token is present
The token is present
The token is present
The token is present
The token is present
The token is present
There is no more tokens in the string

5. Object nextElement()

The nextElement() method is similar to the nextToken() method of the StringTokenizer class;

the difference is that it returns the value in the form of Object unlike the nextToken() method that returns a String.

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class NextElementMethod {
  public static void main(String[] args) throws IOException {
    String myString = "This is TechVidvan Tutorial of Java";
    StringTokenizer st = new StringTokenizer(myString);
    System.out.println("Input string is: " + myString);
    //moving to the next element
    st.nextElement();
    System.out.println("The next element is: " + st.nextElement());
    System.out.println("The next element is: " + st.nextElement());
  }
}

Output:
Input string is: This is TechVidvan Tutorial of Java
The next element is: is
The next element is: TechVidvan

Java StringTokenizer Example

Now, as you are familiar with all the constructors and methods of StringTokenizer class, let’s see an example to work with all of them together in this following example.

Code to understand constructors and methods of StringTokenizer class:

package com.techvidvan.stringtokenizer;
import java.util. * ;
import java.io. * ;
public class StringBufferMethod {
  public static void main(String[] args) throws IOException {
    String myString = "Welcome to StringTokenizer Tutorial, article on StringTokenizer class";
    StringTokenizer st = new StringTokenizer(myString);
    int numberOfTokens;
    numberOfTokens = st.countTokens();
    System.out.println("The number of tokens in this string is: " + numberOfTokens);
    System.out.println(“Printing all the token with Default constructor: ”);
    while (st.hasMoreTokens()) {
      System.out.println(“The next token is” + st.nextToken());
    }
    StringTokenizer st1 = new StringTokenizer(myString, “, ”);
    System.out.println(“Printing all the token with delimiter“, ”: ”);
    while (st1.hasMoreTokens()) {
      System.out.println(st1.nextToken());
    }
    StringTokenizer st2 = new StringTokenizer(myString, “, ”, true);
    System.out.println(“Printing all the token with delimiter“, ”and also printing delimiter: ”);
    while (st2.hasMoreTokens()) {
      System.out.println(st2.nextToken());
    }
  }
}

Output:
The number of tokens in this string is: 8
Printing all the token with Default constructor:
The next token is Welcome
The next token is to
The next token is StringTokenizer
The next token is Tutorial,
The next token is article
The next token is on
The next token is StringTokenizer
The next token is class
Printing all the token with delimiter , :
Welcome to StringTokenizer Tutorial
article on StringTokenizer class
Printing all the token with delimiter , and also printing delimiter :
Welcome to StringTokenizer Tutorial
,
article on StringTokenizer class

Conclusion

We have reached the end of the article. We covered the concept of the String Tokenizer class in Java.

It can be used when we want to work with the words of the String rather than the characters of the string. We learned the constructors and methods of this class with examples so that you can understand them easily.

You can now tokenize your String or break your strings into tokens after going through this article.

Tags:

TechVidvan Team

The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today’s tech industry.